2026-02-13T01:48:51.164 INFO:root:teuthology version: 1.2.4.dev5+g8bec0da71 2026-02-13T01:48:51.180 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-13T01:48:51.255 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/skanta-2026-02-13_01:43:11-rados-wip-bharath17-testing-2026-02-12-0618-tentacle-distro-default-trial/47689 branch: wip-bharath17-testing-2026-02-12-0618-tentacle description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub} email: skanta@redhat.com first_in_suite: false flavor: default job_id: '47689' kernel: branch: distro kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: trial name: skanta-2026-02-13_01:43:11-rados-wip-bharath17-testing-2026-02-12-0618-tentacle-distro-default-trial no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-bharath17-testing-2026-02-12-0618-tentacle ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb workunit: branch: wip-bharath17-testing-2026-02-12-0618-tentacle sha1: 3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb owner: scheduled_skanta@soko04.front.sepia.ceph.com priority: 80 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 2723 sha1: 3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath17-testing-2026-02-12-0618-tentacle suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb targets: trial112.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAsUG1S7pnLINz4XEaV2z/RIeuTtpfgnsDIOAaSt9NW/Wb1RNPjjxpRfohySXp4hGkUlz4orI96+OZE0AJmIpPw= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - scrub teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 8bec0da71becad44414c54979f64c9ef0e7099c6 timestamp: 2026-02-13_01:43:11 tube: trial user: skanta verbose: true worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.3932482 2026-02-13T01:48:51.255 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/qa; will attempt to use it 2026-02-13T01:48:51.256 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/qa/tasks 2026-02-13T01:48:51.256 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-13T01:48:51.257 INFO:teuthology.task.internal:Checking packages... 2026-02-13T01:48:51.257 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb' 2026-02-13T01:48:51.257 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-13T01:48:51.257 INFO:teuthology.packaging:ref: None 2026-02-13T01:48:51.257 INFO:teuthology.packaging:tag: None 2026-02-13T01:48:51.257 INFO:teuthology.packaging:branch: wip-bharath17-testing-2026-02-12-0618-tentacle 2026-02-13T01:48:51.257 INFO:teuthology.packaging:sha1: 3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb 2026-02-13T01:48:51.258 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-bharath17-testing-2026-02-12-0618-tentacle 2026-02-13T01:48:51.376 INFO:teuthology.task.internal:Found packages for ceph version 20.2.0-636-g3b40640f-1jammy 2026-02-13T01:48:51.377 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-13T01:48:51.388 INFO:teuthology.task.internal:no buildpackages task found 2026-02-13T01:48:51.388 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-13T01:48:51.402 INFO:teuthology.task.internal:Saving configuration 2026-02-13T01:48:51.410 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-13T01:48:51.422 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-13T01:48:51.501 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial112.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/skanta-2026-02-13_01:43:11-rados-wip-bharath17-testing-2026-02-12-0618-tentacle-distro-default-trial/47689', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-13 01:45:19.499236', 'locked_by': 'scheduled_skanta@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDyxxG+mKalApwjPeYDqgmiMh/tnMzJ1PbDlN+OFyqlb'} 2026-02-13T01:48:51.501 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-13T01:48:51.512 INFO:teuthology.task.internal:roles: ubuntu@trial112.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-13T01:48:51.512 INFO:teuthology.run_tasks:Running task console_log... 2026-02-13T01:48:51.599 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7955c89db6a0>, signals=[15]) 2026-02-13T01:48:51.599 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-13T01:48:51.610 INFO:teuthology.task.internal:Opening connections... 2026-02-13T01:48:51.610 DEBUG:teuthology.task.internal:connecting to ubuntu@trial112.front.sepia.ceph.com 2026-02-13T01:48:51.611 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-13T01:48:51.681 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-13T01:48:51.693 DEBUG:teuthology.orchestra.run.trial112:> uname -m 2026-02-13T01:48:51.758 INFO:teuthology.orchestra.run.trial112.stdout:x86_64 2026-02-13T01:48:51.758 DEBUG:teuthology.orchestra.run.trial112:> cat /etc/os-release 2026-02-13T01:48:51.807 INFO:teuthology.orchestra.run.trial112.stdout:PRETTY_NAME="Ubuntu 22.04.5 LTS" 2026-02-13T01:48:51.807 INFO:teuthology.orchestra.run.trial112.stdout:NAME="Ubuntu" 2026-02-13T01:48:51.807 INFO:teuthology.orchestra.run.trial112.stdout:VERSION_ID="22.04" 2026-02-13T01:48:51.807 INFO:teuthology.orchestra.run.trial112.stdout:VERSION="22.04.5 LTS (Jammy Jellyfish)" 2026-02-13T01:48:51.807 INFO:teuthology.orchestra.run.trial112.stdout:VERSION_CODENAME=jammy 2026-02-13T01:48:51.808 INFO:teuthology.orchestra.run.trial112.stdout:ID=ubuntu 2026-02-13T01:48:51.808 INFO:teuthology.orchestra.run.trial112.stdout:ID_LIKE=debian 2026-02-13T01:48:51.808 INFO:teuthology.orchestra.run.trial112.stdout:HOME_URL="https://www.ubuntu.com/" 2026-02-13T01:48:51.808 INFO:teuthology.orchestra.run.trial112.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2026-02-13T01:48:51.808 INFO:teuthology.orchestra.run.trial112.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2026-02-13T01:48:51.808 INFO:teuthology.orchestra.run.trial112.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2026-02-13T01:48:51.808 INFO:teuthology.orchestra.run.trial112.stdout:UBUNTU_CODENAME=jammy 2026-02-13T01:48:51.809 INFO:teuthology.lock.ops:Updating trial112.front.sepia.ceph.com on lock server 2026-02-13T01:48:51.899 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-13T01:48:51.917 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-13T01:48:51.931 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-13T01:48:51.931 DEBUG:teuthology.orchestra.run.trial112:> test '!' -e /home/ubuntu/cephtest 2026-02-13T01:48:51.934 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-13T01:48:51.949 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-13T01:48:51.949 DEBUG:teuthology.orchestra.run.trial112:> test -z $(ls -A /var/lib/ceph) 2026-02-13T01:48:51.979 INFO:teuthology.orchestra.run.trial112.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2026-02-13T01:48:51.981 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-13T01:48:52.086 INFO:teuthology.run_tasks:Running task kernel... 2026-02-13T01:48:52.108 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'} 2026-02-13T01:48:52.109 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, timeout 300 2026-02-13T01:48:52.109 DEBUG:teuthology.orchestra.run.trial112:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-13T01:48:52.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-13T01:48:52.112 DEBUG:teuthology.orchestra.run.trial112:> uname -r 2026-02-13T01:48:52.163 INFO:teuthology.orchestra.run.trial112.stdout:5.15.0-164-generic 2026-02-13T01:48:52.163 INFO:teuthology.task.kernel:Running kernel on trial112: 5.15.0-164-generic 2026-02-13T01:48:52.163 DEBUG:teuthology.orchestra.run.trial112:> sudo apt-get clean 2026-02-13T01:48:52.221 DEBUG:teuthology.orchestra.run.trial112:> sudo apt-get update 2026-02-13T01:48:52.680 INFO:teuthology.orchestra.run.trial112.stdout:Hit:1 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-13T01:48:52.755 INFO:teuthology.orchestra.run.trial112.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2026-02-13T01:48:53.038 INFO:teuthology.orchestra.run.trial112.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2026-02-13T01:48:53.162 INFO:teuthology.orchestra.run.trial112.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2026-02-13T01:48:53.281 INFO:teuthology.orchestra.run.trial112.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [3226 kB] 2026-02-13T01:48:53.613 INFO:teuthology.orchestra.run.trial112.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [492 kB] 2026-02-13T01:48:53.623 INFO:teuthology.orchestra.run.trial112.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [19.1 kB] 2026-02-13T01:48:53.624 INFO:teuthology.orchestra.run.trial112.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1254 kB] 2026-02-13T01:48:53.654 INFO:teuthology.orchestra.run.trial112.stdout:Get:9 https://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [313 kB] 2026-02-13T01:48:53.660 INFO:teuthology.orchestra.run.trial112.stdout:Get:10 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [30.2 kB] 2026-02-13T01:48:53.660 INFO:teuthology.orchestra.run.trial112.stdout:Get:11 https://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [5260 kB] 2026-02-13T01:48:53.791 INFO:teuthology.orchestra.run.trial112.stdout:Get:12 https://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [991 kB] 2026-02-13T01:48:53.812 INFO:teuthology.orchestra.run.trial112.stdout:Get:13 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages [2959 kB] 2026-02-13T01:48:53.885 INFO:teuthology.orchestra.run.trial112.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy-security/main Translation-en [425 kB] 2026-02-13T01:48:53.894 INFO:teuthology.orchestra.run.trial112.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [14.1 kB] 2026-02-13T01:48:53.894 INFO:teuthology.orchestra.run.trial112.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [1017 kB] 2026-02-13T01:48:53.920 INFO:teuthology.orchestra.run.trial112.stdout:Get:17 https://archive.ubuntu.com/ubuntu jammy-security/universe Translation-en [224 kB] 2026-02-13T01:48:53.927 INFO:teuthology.orchestra.run.trial112.stdout:Get:18 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [22.6 kB] 2026-02-13T01:48:53.928 INFO:teuthology.orchestra.run.trial112.stdout:Get:19 https://archive.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [5089 kB] 2026-02-13T01:48:54.050 INFO:teuthology.orchestra.run.trial112.stdout:Get:20 https://archive.ubuntu.com/ubuntu jammy-security/restricted Translation-en [963 kB] 2026-02-13T01:48:57.405 INFO:teuthology.orchestra.run.trial112.stdout:Fetched 22.7 MB in 3s (8406 kB/s) 2026-02-13T01:48:57.939 INFO:teuthology.orchestra.run.trial112.stdout:Reading package lists... 2026-02-13T01:48:57.952 DEBUG:teuthology.orchestra.run.trial112:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2026-02-13T01:48:57.986 INFO:teuthology.orchestra.run.trial112.stdout:Reading package lists... 2026-02-13T01:48:58.094 INFO:teuthology.orchestra.run.trial112.stdout:Building dependency tree... 2026-02-13T01:48:58.094 INFO:teuthology.orchestra.run.trial112.stdout:Reading state information... 2026-02-13T01:48:58.184 INFO:teuthology.orchestra.run.trial112.stdout:The following packages were automatically installed and are no longer required: 2026-02-13T01:48:58.184 INFO:teuthology.orchestra.run.trial112.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-13T01:48:58.184 INFO:teuthology.orchestra.run.trial112.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-13T01:48:58.184 INFO:teuthology.orchestra.run.trial112.stdout: sg3-utils-udev 2026-02-13T01:48:58.184 INFO:teuthology.orchestra.run.trial112.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-13T01:48:58.184 INFO:teuthology.orchestra.run.trial112.stdout:The following additional packages will be installed: 2026-02-13T01:48:58.184 INFO:teuthology.orchestra.run.trial112.stdout: linux-generic linux-headers-5.15.0-170 linux-headers-5.15.0-170-generic 2026-02-13T01:48:58.185 INFO:teuthology.orchestra.run.trial112.stdout: linux-headers-generic linux-image-5.15.0-170-generic 2026-02-13T01:48:58.185 INFO:teuthology.orchestra.run.trial112.stdout: linux-modules-5.15.0-170-generic linux-modules-extra-5.15.0-170-generic 2026-02-13T01:48:58.185 INFO:teuthology.orchestra.run.trial112.stdout:Suggested packages: 2026-02-13T01:48:58.185 INFO:teuthology.orchestra.run.trial112.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2026-02-13T01:48:58.198 INFO:teuthology.orchestra.run.trial112.stdout:The following NEW packages will be installed: 2026-02-13T01:48:58.198 INFO:teuthology.orchestra.run.trial112.stdout: linux-headers-5.15.0-170 linux-headers-5.15.0-170-generic 2026-02-13T01:48:58.198 INFO:teuthology.orchestra.run.trial112.stdout: linux-image-5.15.0-170-generic linux-modules-5.15.0-170-generic 2026-02-13T01:48:58.198 INFO:teuthology.orchestra.run.trial112.stdout: linux-modules-extra-5.15.0-170-generic 2026-02-13T01:48:58.198 INFO:teuthology.orchestra.run.trial112.stdout:The following packages will be upgraded: 2026-02-13T01:48:58.198 INFO:teuthology.orchestra.run.trial112.stdout: linux-generic linux-headers-generic linux-image-generic 2026-02-13T01:48:58.293 INFO:teuthology.orchestra.run.trial112.stdout:3 upgraded, 5 newly installed, 0 to remove and 38 not upgraded. 2026-02-13T01:48:58.294 INFO:teuthology.orchestra.run.trial112.stdout:Need to get 113 MB of archives. 2026-02-13T01:48:58.294 INFO:teuthology.orchestra.run.trial112.stdout:After this operation, 585 MB of additional disk space will be used. 2026-02-13T01:48:58.294 INFO:teuthology.orchestra.run.trial112.stdout:Get:1 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-170-generic amd64 5.15.0-170.180 [22.7 MB] 2026-02-13T01:48:58.464 INFO:teuthology.orchestra.run.trial112.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-170-generic amd64 5.15.0-170.180 [11.6 MB] 2026-02-13T01:48:58.509 INFO:teuthology.orchestra.run.trial112.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-170-generic amd64 5.15.0-170.180 [64.0 MB] 2026-02-13T01:48:58.755 INFO:teuthology.orchestra.run.trial112.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.170.159 [1654 B] 2026-02-13T01:48:58.755 INFO:teuthology.orchestra.run.trial112.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.170.159 [2474 B] 2026-02-13T01:48:58.756 INFO:teuthology.orchestra.run.trial112.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-170 all 5.15.0-170.180 [12.4 MB] 2026-02-13T01:48:58.803 INFO:teuthology.orchestra.run.trial112.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-170-generic amd64 5.15.0-170.180 [2828 kB] 2026-02-13T01:48:58.813 INFO:teuthology.orchestra.run.trial112.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.170.159 [2316 B] 2026-02-13T01:48:59.013 INFO:teuthology.orchestra.run.trial112.stdout:Fetched 113 MB in 1s (185 MB/s) 2026-02-13T01:48:59.035 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package linux-modules-5.15.0-170-generic. 2026-02-13T01:48:59.201 INFO:teuthology.orchestra.run.trial112.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 131954 files and directories currently installed.) 2026-02-13T01:48:59.203 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../0-linux-modules-5.15.0-170-generic_5.15.0-170.180_amd64.deb ... 2026-02-13T01:48:59.205 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking linux-modules-5.15.0-170-generic (5.15.0-170.180) ... 2026-02-13T01:48:59.559 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package linux-image-5.15.0-170-generic. 2026-02-13T01:48:59.565 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../1-linux-image-5.15.0-170-generic_5.15.0-170.180_amd64.deb ... 2026-02-13T01:48:59.585 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking linux-image-5.15.0-170-generic (5.15.0-170.180) ... 2026-02-13T01:48:59.626 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-170-generic. 2026-02-13T01:48:59.632 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-170-generic_5.15.0-170.180_amd64.deb ... 2026-02-13T01:48:59.633 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking linux-modules-extra-5.15.0-170-generic (5.15.0-170.180) ... 2026-02-13T01:49:00.747 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../3-linux-generic_5.15.0.170.159_amd64.deb ... 2026-02-13T01:49:00.748 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking linux-generic (5.15.0.170.159) over (5.15.0.164.159) ... 2026-02-13T01:49:00.771 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.170.159_amd64.deb ... 2026-02-13T01:49:00.772 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking linux-image-generic (5.15.0.170.159) over (5.15.0.164.159) ... 2026-02-13T01:49:00.789 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package linux-headers-5.15.0-170. 2026-02-13T01:49:00.796 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../5-linux-headers-5.15.0-170_5.15.0-170.180_all.deb ... 2026-02-13T01:49:00.797 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking linux-headers-5.15.0-170 (5.15.0-170.180) ... 2026-02-13T01:49:02.122 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package linux-headers-5.15.0-170-generic. 2026-02-13T01:49:02.135 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../6-linux-headers-5.15.0-170-generic_5.15.0-170.180_amd64.deb ... 2026-02-13T01:49:02.135 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking linux-headers-5.15.0-170-generic (5.15.0-170.180) ... 2026-02-13T01:49:02.732 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.170.159_amd64.deb ... 2026-02-13T01:49:02.733 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking linux-headers-generic (5.15.0.170.159) over (5.15.0.164.159) ... 2026-02-13T01:49:02.758 INFO:teuthology.orchestra.run.trial112.stdout:Setting up linux-headers-5.15.0-170 (5.15.0-170.180) ... 2026-02-13T01:49:02.759 INFO:teuthology.orchestra.run.trial112.stdout:Setting up linux-headers-5.15.0-170-generic (5.15.0-170.180) ... 2026-02-13T01:49:02.761 INFO:teuthology.orchestra.run.trial112.stdout:Setting up linux-headers-generic (5.15.0.170.159) ... 2026-02-13T01:49:02.762 INFO:teuthology.orchestra.run.trial112.stdout:Setting up linux-modules-5.15.0-170-generic (5.15.0-170.180) ... 2026-02-13T01:49:03.543 INFO:teuthology.orchestra.run.trial112.stdout:Setting up linux-image-5.15.0-170-generic (5.15.0-170.180) ... 2026-02-13T01:49:04.377 INFO:teuthology.orchestra.run.trial112.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.15.0-170-generic 2026-02-13T01:49:04.377 INFO:teuthology.orchestra.run.trial112.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.15.0-170-generic 2026-02-13T01:49:04.383 INFO:teuthology.orchestra.run.trial112.stdout:Setting up linux-modules-extra-5.15.0-170-generic (5.15.0-170.180) ... 2026-02-13T01:49:05.185 INFO:teuthology.orchestra.run.trial112.stdout:Setting up linux-image-generic (5.15.0.170.159) ... 2026-02-13T01:49:05.187 INFO:teuthology.orchestra.run.trial112.stdout:Setting up linux-generic (5.15.0.170.159) ... 2026-02-13T01:49:05.188 INFO:teuthology.orchestra.run.trial112.stdout:Processing triggers for linux-image-5.15.0-170-generic (5.15.0-170.180) ... 2026-02-13T01:49:05.192 INFO:teuthology.orchestra.run.trial112.stdout:/etc/kernel/postinst.d/initramfs-tools: 2026-02-13T01:49:05.192 INFO:teuthology.orchestra.run.trial112.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-170-generic 2026-02-13T01:49:12.269 INFO:teuthology.orchestra.run.trial112.stdout:W: Possible missing firmware /lib/firmware/ast_dp501_fw.bin for module ast 2026-02-13T01:49:18.938 INFO:teuthology.orchestra.run.trial112.stdout:/etc/kernel/postinst.d/zz-update-grub: 2026-02-13T01:49:18.938 INFO:teuthology.orchestra.run.trial112.stdout:Sourcing file `/etc/default/grub' 2026-02-13T01:49:18.957 INFO:teuthology.orchestra.run.trial112.stdout:Sourcing file `/etc/default/grub.d/50-curtin-settings.cfg' 2026-02-13T01:49:18.958 INFO:teuthology.orchestra.run.trial112.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2026-02-13T01:49:18.960 INFO:teuthology.orchestra.run.trial112.stdout:Generating grub configuration file ... 2026-02-13T01:49:19.040 INFO:teuthology.orchestra.run.trial112.stdout:Found linux image: /boot/vmlinuz-5.15.0-170-generic 2026-02-13T01:49:19.047 INFO:teuthology.orchestra.run.trial112.stdout:Found initrd image: /boot/initrd.img-5.15.0-170-generic 2026-02-13T01:49:19.226 INFO:teuthology.orchestra.run.trial112.stdout:Found linux image: /boot/vmlinuz-5.15.0-164-generic 2026-02-13T01:49:19.232 INFO:teuthology.orchestra.run.trial112.stdout:Found initrd image: /boot/initrd.img-5.15.0-164-generic 2026-02-13T01:49:19.306 INFO:teuthology.orchestra.run.trial112.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2026-02-13T01:49:19.306 INFO:teuthology.orchestra.run.trial112.stdout:Systems on them will not be added to the GRUB boot configuration. 2026-02-13T01:49:19.306 INFO:teuthology.orchestra.run.trial112.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2026-02-13T01:49:19.308 INFO:teuthology.orchestra.run.trial112.stdout:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-13T01:49:19.318 INFO:teuthology.orchestra.run.trial112.stdout:done 2026-02-13T01:49:19.724 INFO:teuthology.orchestra.run.trial112.stdout: 2026-02-13T01:49:19.724 INFO:teuthology.orchestra.run.trial112.stdout:Pending kernel upgrade! 2026-02-13T01:49:19.725 INFO:teuthology.orchestra.run.trial112.stdout: 2026-02-13T01:49:19.725 INFO:teuthology.orchestra.run.trial112.stdout:Running kernel version: 2026-02-13T01:49:19.725 INFO:teuthology.orchestra.run.trial112.stdout: 5.15.0-164-generic 2026-02-13T01:49:19.725 INFO:teuthology.orchestra.run.trial112.stdout: 2026-02-13T01:49:19.725 INFO:teuthology.orchestra.run.trial112.stdout:Diagnostics: 2026-02-13T01:49:19.725 INFO:teuthology.orchestra.run.trial112.stdout: The currently running kernel version is not the expected kernel version 5.15.0-170-generic. 2026-02-13T01:49:19.725 INFO:teuthology.orchestra.run.trial112.stdout: 2026-02-13T01:49:19.725 INFO:teuthology.orchestra.run.trial112.stdout:Restarting the system to load the new kernel will not be handled automatically, so you should consider rebooting. [Return] 2026-02-13T01:49:19.725 INFO:teuthology.orchestra.run.trial112.stdout: 2026-02-13T01:49:19.725 INFO:teuthology.orchestra.run.trial112.stdout:The processor microcode seems to be up-to-date. 2026-02-13T01:49:19.725 INFO:teuthology.orchestra.run.trial112.stdout: 2026-02-13T01:49:19.726 INFO:teuthology.orchestra.run.trial112.stdout:No services need to be restarted. 2026-02-13T01:49:19.726 INFO:teuthology.orchestra.run.trial112.stdout: 2026-02-13T01:49:19.726 INFO:teuthology.orchestra.run.trial112.stdout:No containers need to be restarted. 2026-02-13T01:49:19.726 INFO:teuthology.orchestra.run.trial112.stdout: 2026-02-13T01:49:19.726 INFO:teuthology.orchestra.run.trial112.stdout:No user sessions are running outdated binaries. 2026-02-13T01:49:19.726 INFO:teuthology.orchestra.run.trial112.stdout: 2026-02-13T01:49:19.726 INFO:teuthology.orchestra.run.trial112.stdout:No VM guests are running outdated hypervisor (qemu) binaries on this host. 2026-02-13T01:49:20.388 DEBUG:teuthology.orchestra.run.trial112:> dpkg -s linux-image-generic 2026-02-13T01:49:20.398 INFO:teuthology.orchestra.run.trial112.stdout:Package: linux-image-generic 2026-02-13T01:49:20.398 INFO:teuthology.orchestra.run.trial112.stdout:Status: install ok installed 2026-02-13T01:49:20.398 INFO:teuthology.orchestra.run.trial112.stdout:Priority: optional 2026-02-13T01:49:20.398 INFO:teuthology.orchestra.run.trial112.stdout:Section: kernel 2026-02-13T01:49:20.398 INFO:teuthology.orchestra.run.trial112.stdout:Installed-Size: 21 2026-02-13T01:49:20.399 INFO:teuthology.orchestra.run.trial112.stdout:Maintainer: Ubuntu Kernel Team 2026-02-13T01:49:20.399 INFO:teuthology.orchestra.run.trial112.stdout:Architecture: amd64 2026-02-13T01:49:20.399 INFO:teuthology.orchestra.run.trial112.stdout:Source: linux-meta 2026-02-13T01:49:20.399 INFO:teuthology.orchestra.run.trial112.stdout:Version: 5.15.0.170.159 2026-02-13T01:49:20.399 INFO:teuthology.orchestra.run.trial112.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.6), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-170), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.6) 2026-02-13T01:49:20.399 INFO:teuthology.orchestra.run.trial112.stdout:Depends: linux-image-5.15.0-170-generic, linux-modules-extra-5.15.0-170-generic, linux-firmware, intel-microcode, amd64-microcode 2026-02-13T01:49:20.399 INFO:teuthology.orchestra.run.trial112.stdout:Recommends: thermald 2026-02-13T01:49:20.399 INFO:teuthology.orchestra.run.trial112.stdout:Description: Generic Linux kernel image 2026-02-13T01:49:20.399 INFO:teuthology.orchestra.run.trial112.stdout: This package will always depend on the latest generic kernel image 2026-02-13T01:49:20.399 INFO:teuthology.orchestra.run.trial112.stdout: available. 2026-02-13T01:49:20.400 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-164-generic Expected: 5.15.0-170-generic 2026-02-13T01:49:20.400 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-13T01:49:20.400 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-13T01:49:20.400 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2026-02-13T01:49:20.400 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial112.front.sepia.ceph.com, path=None, version=distro) 2026-02-13T01:49:20.400 DEBUG:teuthology.orchestra.run.trial112:> sudo apt-get clean 2026-02-13T01:49:20.480 DEBUG:teuthology.orchestra.run.trial112:> sudo apt-get update 2026-02-13T01:49:20.979 INFO:teuthology.orchestra.run.trial112.stdout:Hit:1 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-13T01:49:21.062 INFO:teuthology.orchestra.run.trial112.stdout:Hit:2 https://archive.ubuntu.com/ubuntu jammy-updates InRelease 2026-02-13T01:49:21.151 INFO:teuthology.orchestra.run.trial112.stdout:Hit:3 https://archive.ubuntu.com/ubuntu jammy-security InRelease 2026-02-13T01:49:21.239 INFO:teuthology.orchestra.run.trial112.stdout:Hit:4 https://archive.ubuntu.com/ubuntu jammy-backports InRelease 2026-02-13T01:49:21.880 INFO:teuthology.orchestra.run.trial112.stdout:Reading package lists... 2026-02-13T01:49:21.893 DEBUG:teuthology.orchestra.run.trial112:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2026-02-13T01:49:21.925 INFO:teuthology.orchestra.run.trial112.stdout:Reading package lists... 2026-02-13T01:49:22.032 INFO:teuthology.orchestra.run.trial112.stdout:Building dependency tree... 2026-02-13T01:49:22.032 INFO:teuthology.orchestra.run.trial112.stdout:Reading state information... 2026-02-13T01:49:22.116 INFO:teuthology.orchestra.run.trial112.stdout:linux-image-generic is already the newest version (5.15.0.170.159). 2026-02-13T01:49:22.116 INFO:teuthology.orchestra.run.trial112.stdout:linux-image-generic set to manually installed. 2026-02-13T01:49:22.116 INFO:teuthology.orchestra.run.trial112.stdout:The following packages were automatically installed and are no longer required: 2026-02-13T01:49:22.116 INFO:teuthology.orchestra.run.trial112.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-13T01:49:22.116 INFO:teuthology.orchestra.run.trial112.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-13T01:49:22.116 INFO:teuthology.orchestra.run.trial112.stdout: sg3-utils-udev 2026-02-13T01:49:22.116 INFO:teuthology.orchestra.run.trial112.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-13T01:49:22.135 INFO:teuthology.orchestra.run.trial112.stdout:0 upgraded, 0 newly installed, 0 to remove and 38 not upgraded. 2026-02-13T01:49:22.138 DEBUG:teuthology.orchestra.run.trial112:> dpkg -s linux-image-generic 2026-02-13T01:49:22.189 INFO:teuthology.orchestra.run.trial112.stdout:Package: linux-image-generic 2026-02-13T01:49:22.189 INFO:teuthology.orchestra.run.trial112.stdout:Status: install ok installed 2026-02-13T01:49:22.189 INFO:teuthology.orchestra.run.trial112.stdout:Priority: optional 2026-02-13T01:49:22.189 INFO:teuthology.orchestra.run.trial112.stdout:Section: kernel 2026-02-13T01:49:22.189 INFO:teuthology.orchestra.run.trial112.stdout:Installed-Size: 21 2026-02-13T01:49:22.190 INFO:teuthology.orchestra.run.trial112.stdout:Maintainer: Ubuntu Kernel Team 2026-02-13T01:49:22.190 INFO:teuthology.orchestra.run.trial112.stdout:Architecture: amd64 2026-02-13T01:49:22.190 INFO:teuthology.orchestra.run.trial112.stdout:Source: linux-meta 2026-02-13T01:49:22.190 INFO:teuthology.orchestra.run.trial112.stdout:Version: 5.15.0.170.159 2026-02-13T01:49:22.190 INFO:teuthology.orchestra.run.trial112.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.6), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-170), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.6) 2026-02-13T01:49:22.190 INFO:teuthology.orchestra.run.trial112.stdout:Depends: linux-image-5.15.0-170-generic, linux-modules-extra-5.15.0-170-generic, linux-firmware, intel-microcode, amd64-microcode 2026-02-13T01:49:22.190 INFO:teuthology.orchestra.run.trial112.stdout:Recommends: thermald 2026-02-13T01:49:22.190 INFO:teuthology.orchestra.run.trial112.stdout:Description: Generic Linux kernel image 2026-02-13T01:49:22.190 INFO:teuthology.orchestra.run.trial112.stdout: This package will always depend on the latest generic kernel image 2026-02-13T01:49:22.190 INFO:teuthology.orchestra.run.trial112.stdout: available. 2026-02-13T01:49:22.192 DEBUG:teuthology.orchestra.run.trial112:> mktemp 2026-02-13T01:49:22.236 INFO:teuthology.orchestra.run.trial112.stdout:/tmp/tmp.6z5hX9Xs1K 2026-02-13T01:49:22.237 DEBUG:teuthology.orchestra.run.trial112:> sudo cp /boot/grub/grub.cfg /tmp/tmp.6z5hX9Xs1K 2026-02-13T01:49:22.289 DEBUG:teuthology.orchestra.run.trial112:> sudo chmod 0666 /tmp/tmp.6z5hX9Xs1K 2026-02-13T01:49:22.386 DEBUG:teuthology.orchestra.remote:trial112:/tmp/tmp.6z5hX9Xs1K is 7KB 2026-02-13T01:49:22.394 DEBUG:teuthology.orchestra.run.trial112:> rm -fr /tmp/tmp.6z5hX9Xs1K 2026-02-13T01:49:22.399 DEBUG:teuthology.orchestra.run.trial112:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2026-02-13T01:49:22.452 DEBUG:teuthology.orchestra.run.trial112:> set -ex 2026-02-13T01:49:22.452 DEBUG:teuthology.orchestra.run.trial112:> sudo dd of=/etc/grub.d/01_ceph_kernel 2026-02-13T01:49:22.453 DEBUG:teuthology.orchestra.run.trial112:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2026-02-13T01:49:22.510 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-170-generic 2026-02-13T01:49:22.511 DEBUG:teuthology.orchestra.run.trial112:> sudo update-grub 2026-02-13T01:49:22.751 INFO:teuthology.orchestra.run.trial112.stderr:Sourcing file `/etc/default/grub' 2026-02-13T01:49:22.769 INFO:teuthology.orchestra.run.trial112.stderr:Sourcing file `/etc/default/grub.d/50-curtin-settings.cfg' 2026-02-13T01:49:22.770 INFO:teuthology.orchestra.run.trial112.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2026-02-13T01:49:22.772 INFO:teuthology.orchestra.run.trial112.stderr:Generating grub configuration file ... 2026-02-13T01:49:22.854 INFO:teuthology.orchestra.run.trial112.stderr:Found linux image: /boot/vmlinuz-5.15.0-170-generic 2026-02-13T01:49:22.860 INFO:teuthology.orchestra.run.trial112.stderr:Found initrd image: /boot/initrd.img-5.15.0-170-generic 2026-02-13T01:49:23.032 INFO:teuthology.orchestra.run.trial112.stderr:Found linux image: /boot/vmlinuz-5.15.0-164-generic 2026-02-13T01:49:23.038 INFO:teuthology.orchestra.run.trial112.stderr:Found initrd image: /boot/initrd.img-5.15.0-164-generic 2026-02-13T01:49:23.110 INFO:teuthology.orchestra.run.trial112.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2026-02-13T01:49:23.110 INFO:teuthology.orchestra.run.trial112.stderr:Systems on them will not be added to the GRUB boot configuration. 2026-02-13T01:49:23.110 INFO:teuthology.orchestra.run.trial112.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2026-02-13T01:49:23.113 INFO:teuthology.orchestra.run.trial112.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-13T01:49:23.120 INFO:teuthology.orchestra.run.trial112.stderr:done 2026-02-13T01:49:23.122 DEBUG:teuthology.orchestra.run.trial112:> sudo shutdown -r now 2026-02-13T01:49:53.154 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2026-02-13T01:49:53.154 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial112.front.sepia.ceph.com' 2026-02-13T01:49:53.155 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-13T01:50:53.161 DEBUG:teuthology.orchestra.remote:timed out 2026-02-13T01:51:02.169 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial112.front.sepia.ceph.com' 2026-02-13T01:51:02.170 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-13T01:51:02.262 DEBUG:teuthology.orchestra.run.trial112:> true 2026-02-13T01:51:02.670 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial112.front.sepia.ceph.com' 2026-02-13T01:51:02.670 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-170-generic"... 2026-02-13T01:51:02.670 DEBUG:teuthology.orchestra.run.trial112:> uname -r 2026-02-13T01:51:02.714 INFO:teuthology.orchestra.run.trial112.stdout:5.15.0-170-generic 2026-02-13T01:51:02.715 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-170-generic vs 5.15.0-170-generic 2026-02-13T01:51:02.715 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-13T01:51:02.715 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2026-02-13T01:51:03.716 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-13T01:51:03.716 DEBUG:teuthology.orchestra.run.trial112:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-13T01:51:03.729 INFO:teuthology.orchestra.run.trial112.stdout:ttyS1 2026-02-13T01:51:03.752 DEBUG:teuthology.parallel:result is None 2026-02-13T01:51:03.752 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-13T01:51:03.764 INFO:teuthology.task.internal:Creating test directory... 2026-02-13T01:51:03.764 DEBUG:teuthology.orchestra.run.trial112:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-13T01:51:03.797 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-13T01:51:03.809 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-13T01:51:03.823 INFO:teuthology.task.internal:Creating archive directory... 2026-02-13T01:51:03.823 DEBUG:teuthology.orchestra.run.trial112:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-13T01:51:03.843 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-13T01:51:03.854 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-13T01:51:03.854 DEBUG:teuthology.orchestra.run.trial112:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-13T01:51:03.890 INFO:teuthology.orchestra.run.trial112.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T01:51:03.896 INFO:teuthology.orchestra.run.trial112.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T01:51:03.897 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-13T01:51:03.909 INFO:teuthology.task.internal:Configuring sudo... 2026-02-13T01:51:03.910 DEBUG:teuthology.orchestra.run.trial112:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-13T01:51:03.945 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-13T01:51:03.958 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-13T01:51:03.958 DEBUG:teuthology.orchestra.run.trial112:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-13T01:51:03.993 DEBUG:teuthology.orchestra.run.trial112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-13T01:51:04.038 DEBUG:teuthology.orchestra.run.trial112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-13T01:51:04.081 DEBUG:teuthology.orchestra.run.trial112:> set -ex 2026-02-13T01:51:04.082 DEBUG:teuthology.orchestra.run.trial112:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-13T01:51:04.131 DEBUG:teuthology.orchestra.run.trial112:> sudo service rsyslog restart 2026-02-13T01:51:04.191 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-13T01:51:04.204 INFO:teuthology.task.internal:Starting timer... 2026-02-13T01:51:04.204 INFO:teuthology.run_tasks:Running task pcp... 2026-02-13T01:51:04.226 INFO:teuthology.run_tasks:Running task selinux... 2026-02-13T01:51:04.244 DEBUG:teuthology.task.selinux:Excluding trial112: OS 'ubuntu' does not support SELinux 2026-02-13T01:51:04.244 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-13T01:51:04.244 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2026-02-13T01:51:04.244 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-13T01:51:04.245 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-13T01:51:04.260 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-13T01:51:04.272 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-13T01:51:04.274 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial112.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-13T01:52:02.159 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial112.front.sepia.ceph.com')] 2026-02-13T01:52:02.162 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial112.front.sepia.ceph.com' 2026-02-13T01:52:02.164 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-13T01:52:02.283 DEBUG:teuthology.orchestra.run.trial112:> true 2026-02-13T01:52:02.363 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial112.front.sepia.ceph.com' 2026-02-13T01:52:02.363 INFO:teuthology.run_tasks:Running task clock... 2026-02-13T01:52:02.381 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-13T01:52:02.382 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-13T01:52:02.382 DEBUG:teuthology.orchestra.run.trial112:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-13T01:52:02.422 INFO:teuthology.orchestra.run.trial112.stdout:13 Feb 01:52:02 ntpd[4885]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2026-02-13T01:52:02.422 INFO:teuthology.orchestra.run.trial112.stdout:13 Feb 01:52:02 ntpd[4885]: Command line: ntpd -gq 2026-02-13T01:52:02.422 INFO:teuthology.orchestra.run.trial112.stdout:13 Feb 01:52:02 ntpd[4885]: ---------------------------------------------------- 2026-02-13T01:52:02.422 INFO:teuthology.orchestra.run.trial112.stdout:13 Feb 01:52:02 ntpd[4885]: ntp-4 is maintained by Network Time Foundation, 2026-02-13T01:52:02.422 INFO:teuthology.orchestra.run.trial112.stdout:13 Feb 01:52:02 ntpd[4885]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2026-02-13T01:52:02.422 INFO:teuthology.orchestra.run.trial112.stdout:13 Feb 01:52:02 ntpd[4885]: corporation. Support and training for ntp-4 are 2026-02-13T01:52:02.423 INFO:teuthology.orchestra.run.trial112.stdout:13 Feb 01:52:02 ntpd[4885]: available at https://www.nwtime.org/support 2026-02-13T01:52:02.423 INFO:teuthology.orchestra.run.trial112.stdout:13 Feb 01:52:02 ntpd[4885]: ---------------------------------------------------- 2026-02-13T01:52:02.423 INFO:teuthology.orchestra.run.trial112.stdout:13 Feb 01:52:02 ntpd[4885]: proto: precision = 0.020 usec (-25) 2026-02-13T01:52:02.423 INFO:teuthology.orchestra.run.trial112.stdout:13 Feb 01:52:02 ntpd[4885]: basedate set to 2022-02-04 2026-02-13T01:52:02.423 INFO:teuthology.orchestra.run.trial112.stdout:13 Feb 01:52:02 ntpd[4885]: gps base set to 2022-02-06 (week 2196) 2026-02-13T01:52:02.423 INFO:teuthology.orchestra.run.trial112.stdout:13 Feb 01:52:02 ntpd[4885]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-13T01:52:02.423 INFO:teuthology.orchestra.run.trial112.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-13T01:52:02.423 INFO:teuthology.orchestra.run.trial112.stderr:restrict ::: KOD does nothing without LIMITED. 2026-02-13T01:52:02.424 INFO:teuthology.orchestra.run.trial112.stdout:13 Feb 01:52:02 ntpd[4885]: restrict ::: KOD does nothing without LIMITED. 2026-02-13T01:52:02.424 INFO:teuthology.orchestra.run.trial112.stdout:13 Feb 01:52:02 ntpd[4885]: Listen and drop on 0 v6wildcard [::]:123 2026-02-13T01:52:02.424 INFO:teuthology.orchestra.run.trial112.stdout:13 Feb 01:52:02 ntpd[4885]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2026-02-13T01:52:02.424 INFO:teuthology.orchestra.run.trial112.stdout:13 Feb 01:52:02 ntpd[4885]: Listen normally on 2 lo 127.0.0.1:123 2026-02-13T01:52:02.424 INFO:teuthology.orchestra.run.trial112.stdout:13 Feb 01:52:02 ntpd[4885]: Listen normally on 3 enp1s0f0 10.20.193.112:123 2026-02-13T01:52:02.424 INFO:teuthology.orchestra.run.trial112.stdout:13 Feb 01:52:02 ntpd[4885]: Listen normally on 4 lo [::1]:123 2026-02-13T01:52:02.424 INFO:teuthology.orchestra.run.trial112.stdout:13 Feb 01:52:02 ntpd[4885]: Listen normally on 5 enp1s0f0 [fe80::925a:8ff:fe77:5df0%2]:123 2026-02-13T01:52:02.424 INFO:teuthology.orchestra.run.trial112.stdout:13 Feb 01:52:02 ntpd[4885]: Listening on routing socket on fd #22 for interface updates 2026-02-13T01:52:03.422 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:03 ntpd[4885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-13T01:52:03.422 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:03 ntpd[4885]: can't open /var/log/ntpstats/rawstats.20260213: Permission denied 2026-02-13T01:52:03.422 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:03 ntpd[4885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-13T01:52:03.423 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:03 ntpd[4885]: can't open /var/log/ntpstats/rawstats.20260213: Permission denied 2026-02-13T01:52:03.433 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:03 ntpd[4885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-13T01:52:03.433 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:03 ntpd[4885]: can't open /var/log/ntpstats/rawstats.20260213: Permission denied 2026-02-13T01:52:03.433 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:03 ntpd[4885]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-13T01:52:03.433 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:03 ntpd[4885]: can't open /var/log/ntpstats/peerstats.20260213: Permission denied 2026-02-13T01:52:03.444 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:03 ntpd[4885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-13T01:52:03.445 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:03 ntpd[4885]: can't open /var/log/ntpstats/rawstats.20260213: Permission denied 2026-02-13T01:52:03.445 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:03 ntpd[4885]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-13T01:52:03.445 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:03 ntpd[4885]: can't open /var/log/ntpstats/peerstats.20260213: Permission denied 2026-02-13T01:52:05.423 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:05 ntpd[4885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-13T01:52:05.423 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:05 ntpd[4885]: can't open /var/log/ntpstats/rawstats.20260213: Permission denied 2026-02-13T01:52:05.423 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:05 ntpd[4885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-13T01:52:05.424 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:05 ntpd[4885]: can't open /var/log/ntpstats/rawstats.20260213: Permission denied 2026-02-13T01:52:05.424 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:05 ntpd[4885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-13T01:52:05.424 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:05 ntpd[4885]: can't open /var/log/ntpstats/rawstats.20260213: Permission denied 2026-02-13T01:52:05.433 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:05 ntpd[4885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-13T01:52:05.433 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:05 ntpd[4885]: can't open /var/log/ntpstats/rawstats.20260213: Permission denied 2026-02-13T01:52:05.433 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:05 ntpd[4885]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-13T01:52:05.433 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:05 ntpd[4885]: can't open /var/log/ntpstats/peerstats.20260213: Permission denied 2026-02-13T01:52:05.443 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:05 ntpd[4885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-13T01:52:05.443 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:05 ntpd[4885]: can't open /var/log/ntpstats/rawstats.20260213: Permission denied 2026-02-13T01:52:05.444 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:05 ntpd[4885]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-13T01:52:05.444 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:05 ntpd[4885]: can't open /var/log/ntpstats/peerstats.20260213: Permission denied 2026-02-13T01:52:05.460 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:05 ntpd[4885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-13T01:52:05.460 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:05 ntpd[4885]: can't open /var/log/ntpstats/rawstats.20260213: Permission denied 2026-02-13T01:52:05.460 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:05 ntpd[4885]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-13T01:52:05.460 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:05 ntpd[4885]: can't open /var/log/ntpstats/peerstats.20260213: Permission denied 2026-02-13T01:52:07.423 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:07 ntpd[4885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-13T01:52:07.423 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:07 ntpd[4885]: can't open /var/log/ntpstats/rawstats.20260213: Permission denied 2026-02-13T01:52:07.423 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:07 ntpd[4885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-13T01:52:07.423 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:07 ntpd[4885]: can't open /var/log/ntpstats/rawstats.20260213: Permission denied 2026-02-13T01:52:07.423 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:07 ntpd[4885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-13T01:52:07.423 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:07 ntpd[4885]: can't open /var/log/ntpstats/rawstats.20260213: Permission denied 2026-02-13T01:52:07.423 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:07 ntpd[4885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-13T01:52:07.423 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:07 ntpd[4885]: can't open /var/log/ntpstats/rawstats.20260213: Permission denied 2026-02-13T01:52:07.432 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:07 ntpd[4885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-13T01:52:07.433 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:07 ntpd[4885]: can't open /var/log/ntpstats/rawstats.20260213: Permission denied 2026-02-13T01:52:07.433 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:07 ntpd[4885]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-13T01:52:07.433 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:07 ntpd[4885]: can't open /var/log/ntpstats/peerstats.20260213: Permission denied 2026-02-13T01:52:07.435 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:07 ntpd[4885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-13T01:52:07.435 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:07 ntpd[4885]: can't open /var/log/ntpstats/rawstats.20260213: Permission denied 2026-02-13T01:52:07.435 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:07 ntpd[4885]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-13T01:52:07.435 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:07 ntpd[4885]: can't open /var/log/ntpstats/peerstats.20260213: Permission denied 2026-02-13T01:52:07.459 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:07 ntpd[4885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-13T01:52:07.459 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:07 ntpd[4885]: can't open /var/log/ntpstats/rawstats.20260213: Permission denied 2026-02-13T01:52:07.459 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:07 ntpd[4885]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-13T01:52:07.460 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:07 ntpd[4885]: can't open /var/log/ntpstats/peerstats.20260213: Permission denied 2026-02-13T01:52:07.463 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:07 ntpd[4885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-13T01:52:07.463 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:07 ntpd[4885]: can't open /var/log/ntpstats/rawstats.20260213: Permission denied 2026-02-13T01:52:07.463 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:07 ntpd[4885]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-13T01:52:07.464 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:07 ntpd[4885]: can't open /var/log/ntpstats/peerstats.20260213: Permission denied 2026-02-13T01:52:09.423 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:09 ntpd[4885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-13T01:52:09.423 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:09 ntpd[4885]: can't open /var/log/ntpstats/rawstats.20260213: Permission denied 2026-02-13T01:52:09.423 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:09 ntpd[4885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-13T01:52:09.423 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:09 ntpd[4885]: can't open /var/log/ntpstats/rawstats.20260213: Permission denied 2026-02-13T01:52:09.423 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:09 ntpd[4885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-13T01:52:09.423 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:09 ntpd[4885]: can't open /var/log/ntpstats/rawstats.20260213: Permission denied 2026-02-13T01:52:09.423 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:09 ntpd[4885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-13T01:52:09.423 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:09 ntpd[4885]: can't open /var/log/ntpstats/rawstats.20260213: Permission denied 2026-02-13T01:52:09.433 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:09 ntpd[4885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-13T01:52:09.433 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:09 ntpd[4885]: can't open /var/log/ntpstats/rawstats.20260213: Permission denied 2026-02-13T01:52:09.433 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:09 ntpd[4885]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-13T01:52:09.433 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:09 ntpd[4885]: can't open /var/log/ntpstats/peerstats.20260213: Permission denied 2026-02-13T01:52:09.433 INFO:teuthology.orchestra.run.trial112.stdout:13 Feb 01:52:09 ntpd[4885]: ntpd: time set +0.159512 s 2026-02-13T01:52:09.433 INFO:teuthology.orchestra.run.trial112.stdout:ntpd: time set +0.159512s 2026-02-13T01:52:09.433 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:09 ntpd[4885]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2026-02-13T01:52:09.434 INFO:teuthology.orchestra.run.trial112.stderr:13 Feb 01:52:09 ntpd[4885]: can't open /var/log/ntpstats/loopstats.20260213: Permission denied 2026-02-13T01:52:09.456 INFO:teuthology.orchestra.run.trial112.stdout:No association ID's returned 2026-02-13T01:52:09.456 INFO:teuthology.run_tasks:Running task install... 2026-02-13T01:52:09.468 DEBUG:teuthology.task.install:project ceph 2026-02-13T01:52:09.468 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb'}} 2026-02-13T01:52:09.468 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb'} 2026-02-13T01:52:09.468 INFO:teuthology.task.install:Using flavor: default 2026-02-13T01:52:09.470 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-13T01:52:09.470 INFO:teuthology.task.install:extra packages: [] 2026-02-13T01:52:09.471 DEBUG:teuthology.orchestra.run.trial112:> sudo apt-key list | grep Ceph 2026-02-13T01:52:09.553 INFO:teuthology.orchestra.run.trial112.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2026-02-13T01:52:09.583 INFO:teuthology.orchestra.run.trial112.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2026-02-13T01:52:09.583 INFO:teuthology.orchestra.run.trial112.stdout:uid [ unknown] Ceph.com (release key) 2026-02-13T01:52:09.584 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2026-02-13T01:52:09.584 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb 2026-02-13T01:52:09.724 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-bharath17-testing-2026-02-12-0618-tentacle/3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/ubuntu/jammy/flavors/default/ 2026-02-13T01:52:09.724 INFO:teuthology.task.install.deb:Package version is 20.2.0-636-g3b40640f-1jammy 2026-02-13T01:52:10.057 DEBUG:teuthology.orchestra.run.trial112:> set -ex 2026-02-13T01:52:10.057 DEBUG:teuthology.orchestra.run.trial112:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2026-02-13T01:52:10.067 DEBUG:teuthology.orchestra.run.trial112:> sudo apt-get update 2026-02-13T01:52:10.446 INFO:teuthology.orchestra.run.trial112.stdout:Ign:1 https://1.chacra.ceph.com/r/ceph/wip-bharath17-testing-2026-02-12-0618-tentacle/3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/ubuntu/jammy/flavors/default jammy InRelease 2026-02-13T01:52:10.502 INFO:teuthology.orchestra.run.trial112.stdout:Hit:2 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-13T01:52:10.516 INFO:teuthology.orchestra.run.trial112.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/wip-bharath17-testing-2026-02-12-0618-tentacle/3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/ubuntu/jammy/flavors/default jammy Release [7680 B] 2026-02-13T01:52:10.568 INFO:teuthology.orchestra.run.trial112.stdout:Hit:4 https://archive.ubuntu.com/ubuntu jammy-updates InRelease 2026-02-13T01:52:10.587 INFO:teuthology.orchestra.run.trial112.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/wip-bharath17-testing-2026-02-12-0618-tentacle/3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/ubuntu/jammy/flavors/default jammy Release.gpg 2026-02-13T01:52:10.640 INFO:teuthology.orchestra.run.trial112.stdout:Hit:6 https://archive.ubuntu.com/ubuntu jammy-security InRelease 2026-02-13T01:52:10.658 INFO:teuthology.orchestra.run.trial112.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-bharath17-testing-2026-02-12-0618-tentacle/3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.8 kB] 2026-02-13T01:52:10.712 INFO:teuthology.orchestra.run.trial112.stdout:Hit:8 https://archive.ubuntu.com/ubuntu jammy-backports InRelease 2026-02-13T01:52:10.857 INFO:teuthology.orchestra.run.trial112.stdout:Fetched 26.5 kB in 1s (42.1 kB/s) 2026-02-13T01:52:11.388 INFO:teuthology.orchestra.run.trial112.stdout:Reading package lists... 2026-02-13T01:52:11.402 DEBUG:teuthology.orchestra.run.trial112:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=20.2.0-636-g3b40640f-1jammy cephadm=20.2.0-636-g3b40640f-1jammy ceph-mds=20.2.0-636-g3b40640f-1jammy ceph-mgr=20.2.0-636-g3b40640f-1jammy ceph-common=20.2.0-636-g3b40640f-1jammy ceph-fuse=20.2.0-636-g3b40640f-1jammy ceph-test=20.2.0-636-g3b40640f-1jammy ceph-volume=20.2.0-636-g3b40640f-1jammy radosgw=20.2.0-636-g3b40640f-1jammy python3-rados=20.2.0-636-g3b40640f-1jammy python3-rgw=20.2.0-636-g3b40640f-1jammy python3-cephfs=20.2.0-636-g3b40640f-1jammy python3-rbd=20.2.0-636-g3b40640f-1jammy libcephfs2=20.2.0-636-g3b40640f-1jammy libcephfs-dev=20.2.0-636-g3b40640f-1jammy librados2=20.2.0-636-g3b40640f-1jammy librbd1=20.2.0-636-g3b40640f-1jammy rbd-fuse=20.2.0-636-g3b40640f-1jammy 2026-02-13T01:52:11.436 INFO:teuthology.orchestra.run.trial112.stdout:Reading package lists... 2026-02-13T01:52:11.548 INFO:teuthology.orchestra.run.trial112.stdout:Building dependency tree... 2026-02-13T01:52:11.548 INFO:teuthology.orchestra.run.trial112.stdout:Reading state information... 2026-02-13T01:52:11.639 INFO:teuthology.orchestra.run.trial112.stdout:The following packages were automatically installed and are no longer required: 2026-02-13T01:52:11.639 INFO:teuthology.orchestra.run.trial112.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-13T01:52:11.639 INFO:teuthology.orchestra.run.trial112.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-13T01:52:11.639 INFO:teuthology.orchestra.run.trial112.stdout: sg3-utils-udev 2026-02-13T01:52:11.639 INFO:teuthology.orchestra.run.trial112.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-13T01:52:11.640 INFO:teuthology.orchestra.run.trial112.stdout:The following additional packages will be installed: 2026-02-13T01:52:11.640 INFO:teuthology.orchestra.run.trial112.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2026-02-13T01:52:11.640 INFO:teuthology.orchestra.run.trial112.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2026-02-13T01:52:11.640 INFO:teuthology.orchestra.run.trial112.stdout: libcephfs-daemon libcephfs-proxy2 libdouble-conversion3 libfuse2 libjq1 2026-02-13T01:52:11.640 INFO:teuthology.orchestra.run.trial112.stdout: liblttng-ust1 libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2026-02-13T01:52:11.640 INFO:teuthology.orchestra.run.trial112.stdout: libqt5dbus5 libqt5network5 libradosstriper1 librdkafka1 librgw2 2026-02-13T01:52:11.640 INFO:teuthology.orchestra.run.trial112.stdout: libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli python-asyncssh-doc 2026-02-13T01:52:11.640 INFO:teuthology.orchestra.run.trial112.stdout: python3-asyncssh python3-cachetools python3-ceph-argparse 2026-02-13T01:52:11.640 INFO:teuthology.orchestra.run.trial112.stdout: python3-ceph-common python3-cheroot python3-cherrypy3 python3-google-auth 2026-02-13T01:52:11.640 INFO:teuthology.orchestra.run.trial112.stdout: python3-iniconfig python3-jaraco.classes python3-jaraco.collections 2026-02-13T01:52:11.640 INFO:teuthology.orchestra.run.trial112.stdout: python3-jaraco.functools python3-jaraco.text python3-joblib 2026-02-13T01:52:11.640 INFO:teuthology.orchestra.run.trial112.stdout: python3-kubernetes python3-natsort python3-pluggy python3-portend 2026-02-13T01:52:11.640 INFO:teuthology.orchestra.run.trial112.stdout: python3-prettytable python3-psutil python3-py python3-pygments 2026-02-13T01:52:11.640 INFO:teuthology.orchestra.run.trial112.stdout: python3-pytest python3-repoze.lru python3-requests-oauthlib python3-routes 2026-02-13T01:52:11.640 INFO:teuthology.orchestra.run.trial112.stdout: python3-rsa python3-simplejson python3-sklearn python3-sklearn-lib 2026-02-13T01:52:11.640 INFO:teuthology.orchestra.run.trial112.stdout: python3-tempora python3-threadpoolctl python3-toml python3-wcwidth 2026-02-13T01:52:11.640 INFO:teuthology.orchestra.run.trial112.stdout: python3-webob python3-websocket python3-zc.lockfile qttranslations5-l10n 2026-02-13T01:52:11.640 INFO:teuthology.orchestra.run.trial112.stdout: socat xmlstarlet 2026-02-13T01:52:11.640 INFO:teuthology.orchestra.run.trial112.stdout:Suggested packages: 2026-02-13T01:52:11.640 INFO:teuthology.orchestra.run.trial112.stdout: python3-influxdb liblua5.3-dev luarocks python-natsort-doc python-psutil-doc 2026-02-13T01:52:11.640 INFO:teuthology.orchestra.run.trial112.stdout: subversion python-pygments-doc ttf-bitstream-vera python3-paste python3-dap 2026-02-13T01:52:11.640 INFO:teuthology.orchestra.run.trial112.stdout: python-sklearn-doc ipython3 python-webob-doc 2026-02-13T01:52:11.640 INFO:teuthology.orchestra.run.trial112.stdout:Recommended packages: 2026-02-13T01:52:11.640 INFO:teuthology.orchestra.run.trial112.stdout: btrfs-tools 2026-02-13T01:52:11.673 INFO:teuthology.orchestra.run.trial112.stdout:The following NEW packages will be installed: 2026-02-13T01:52:11.674 INFO:teuthology.orchestra.run.trial112.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2026-02-13T01:52:11.674 INFO:teuthology.orchestra.run.trial112.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2026-02-13T01:52:11.674 INFO:teuthology.orchestra.run.trial112.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2026-02-13T01:52:11.674 INFO:teuthology.orchestra.run.trial112.stdout: libcephfs-daemon libcephfs-dev libcephfs-proxy2 libcephfs2 2026-02-13T01:52:11.674 INFO:teuthology.orchestra.run.trial112.stdout: libdouble-conversion3 libfuse2 libjq1 liblttng-ust1 libnbd0 liboath0 2026-02-13T01:52:11.674 INFO:teuthology.orchestra.run.trial112.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2026-02-13T01:52:11.674 INFO:teuthology.orchestra.run.trial112.stdout: libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2026-02-13T01:52:11.674 INFO:teuthology.orchestra.run.trial112.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python3-asyncssh 2026-02-13T01:52:11.674 INFO:teuthology.orchestra.run.trial112.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2026-02-13T01:52:11.674 INFO:teuthology.orchestra.run.trial112.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-iniconfig 2026-02-13T01:52:11.674 INFO:teuthology.orchestra.run.trial112.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2026-02-13T01:52:11.674 INFO:teuthology.orchestra.run.trial112.stdout: python3-jaraco.text python3-joblib python3-kubernetes python3-natsort 2026-02-13T01:52:11.674 INFO:teuthology.orchestra.run.trial112.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2026-02-13T01:52:11.674 INFO:teuthology.orchestra.run.trial112.stdout: python3-pygments python3-pytest python3-rados python3-rbd python3-repoze.lru 2026-02-13T01:52:11.674 INFO:teuthology.orchestra.run.trial112.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2026-02-13T01:52:11.674 INFO:teuthology.orchestra.run.trial112.stdout: python3-simplejson python3-sklearn python3-sklearn-lib python3-tempora 2026-02-13T01:52:11.674 INFO:teuthology.orchestra.run.trial112.stdout: python3-threadpoolctl python3-toml python3-wcwidth python3-webob 2026-02-13T01:52:11.674 INFO:teuthology.orchestra.run.trial112.stdout: python3-websocket python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2026-02-13T01:52:11.674 INFO:teuthology.orchestra.run.trial112.stdout: socat xmlstarlet 2026-02-13T01:52:12.017 INFO:teuthology.orchestra.run.trial112.stdout:0 upgraded, 86 newly installed, 0 to remove and 38 not upgraded. 2026-02-13T01:52:12.018 INFO:teuthology.orchestra.run.trial112.stdout:Need to get 280 MB of archives. 2026-02-13T01:52:12.018 INFO:teuthology.orchestra.run.trial112.stdout:After this operation, 1119 MB of additional disk space will be used. 2026-02-13T01:52:12.018 INFO:teuthology.orchestra.run.trial112.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/wip-bharath17-testing-2026-02-12-0618-tentacle/3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 20.2.0-636-g3b40640f-1jammy [3574 kB] 2026-02-13T01:52:12.029 INFO:teuthology.orchestra.run.trial112.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2026-02-13T01:52:12.367 INFO:teuthology.orchestra.run.trial112.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2026-02-13T01:52:12.377 INFO:teuthology.orchestra.run.trial112.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2026-02-13T01:52:12.447 INFO:teuthology.orchestra.run.trial112.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2006 kB] 2026-02-13T01:52:12.649 INFO:teuthology.orchestra.run.trial112.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2026-02-13T01:52:12.659 INFO:teuthology.orchestra.run.trial112.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2026-02-13T01:52:12.686 INFO:teuthology.orchestra.run.trial112.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2026-02-13T01:52:12.694 INFO:teuthology.orchestra.run.trial112.stdout:Get:9 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2026-02-13T01:52:12.696 INFO:teuthology.orchestra.run.trial112.stdout:Get:10 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2026-02-13T01:52:12.697 INFO:teuthology.orchestra.run.trial112.stdout:Get:11 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2026-02-13T01:52:12.697 INFO:teuthology.orchestra.run.trial112.stdout:Get:12 https://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2026-02-13T01:52:12.714 INFO:teuthology.orchestra.run.trial112.stdout:Get:13 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 liboath0 amd64 2.6.7-3ubuntu0.1 [41.3 kB] 2026-02-13T01:52:12.715 INFO:teuthology.orchestra.run.trial112.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9030 B] 2026-02-13T01:52:12.721 INFO:teuthology.orchestra.run.trial112.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2026-02-13T01:52:12.788 INFO:teuthology.orchestra.run.trial112.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6452 B] 2026-02-13T01:52:12.788 INFO:teuthology.orchestra.run.trial112.stdout:Get:17 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8716 B] 2026-02-13T01:52:12.788 INFO:teuthology.orchestra.run.trial112.stdout:Get:18 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2026-02-13T01:52:12.788 INFO:teuthology.orchestra.run.trial112.stdout:Get:19 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2026-02-13T01:52:12.789 INFO:teuthology.orchestra.run.trial112.stdout:Get:20 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7240 B] 2026-02-13T01:52:12.789 INFO:teuthology.orchestra.run.trial112.stdout:Get:21 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8980 B] 2026-02-13T01:52:12.789 INFO:teuthology.orchestra.run.trial112.stdout:Get:22 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2026-02-13T01:52:12.793 INFO:teuthology.orchestra.run.trial112.stdout:Get:23 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2026-02-13T01:52:12.794 INFO:teuthology.orchestra.run.trial112.stdout:Get:24 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libfuse2 amd64 2.9.9-5ubuntu3 [90.3 kB] 2026-02-13T01:52:12.823 INFO:teuthology.orchestra.run.trial112.stdout:Get:25 https://1.chacra.ceph.com/r/ceph/wip-bharath17-testing-2026-02-12-0618-tentacle/3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 20.2.0-636-g3b40640f-1jammy [2866 kB] 2026-02-13T01:52:12.864 INFO:teuthology.orchestra.run.trial112.stdout:Get:26 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python3-asyncssh all 2.5.0-1ubuntu0.1 [189 kB] 2026-02-13T01:52:12.865 INFO:teuthology.orchestra.run.trial112.stdout:Get:27 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2026-02-13T01:52:12.866 INFO:teuthology.orchestra.run.trial112.stdout:Get:28 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2026-02-13T01:52:12.867 INFO:teuthology.orchestra.run.trial112.stdout:Get:29 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2058 kB] 2026-02-13T01:52:12.892 INFO:teuthology.orchestra.run.trial112.stdout:Get:30 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2026-02-13T01:52:12.894 INFO:teuthology.orchestra.run.trial112.stdout:Get:31 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2026-02-13T01:52:12.894 INFO:teuthology.orchestra.run.trial112.stdout:Get:32 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1829 kB] 2026-02-13T01:52:12.952 INFO:teuthology.orchestra.run.trial112.stdout:Get:33 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9722 B] 2026-02-13T01:52:12.952 INFO:teuthology.orchestra.run.trial112.stdout:Get:34 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2026-02-13T01:52:12.952 INFO:teuthology.orchestra.run.trial112.stdout:Get:35 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2026-02-13T01:52:12.953 INFO:teuthology.orchestra.run.trial112.stdout:Get:36 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2026-02-13T01:52:12.953 INFO:teuthology.orchestra.run.trial112.stdout:Get:37 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2026-02-13T01:52:12.953 INFO:teuthology.orchestra.run.trial112.stdout:Get:38 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2026-02-13T01:52:13.012 INFO:teuthology.orchestra.run.trial112.stdout:Get:39 https://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2026-02-13T01:52:13.014 INFO:teuthology.orchestra.run.trial112.stdout:Get:40 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libjq1 amd64 1.6-2.1ubuntu3.1 [133 kB] 2026-02-13T01:52:13.015 INFO:teuthology.orchestra.run.trial112.stdout:Get:41 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 jq amd64 1.6-2.1ubuntu3.1 [52.5 kB] 2026-02-13T01:52:13.085 INFO:teuthology.orchestra.run.trial112.stdout:Get:42 https://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2026-02-13T01:52:13.087 INFO:teuthology.orchestra.run.trial112.stdout:Get:43 https://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2026-02-13T01:52:13.090 INFO:teuthology.orchestra.run.trial112.stdout:Get:44 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2026-02-13T01:52:13.093 INFO:teuthology.orchestra.run.trial112.stdout:Get:45 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python-asyncssh-doc all 2.5.0-1ubuntu0.1 [309 kB] 2026-02-13T01:52:13.096 INFO:teuthology.orchestra.run.trial112.stdout:Get:46 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6024 B] 2026-02-13T01:52:13.096 INFO:teuthology.orchestra.run.trial112.stdout:Get:47 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2026-02-13T01:52:13.096 INFO:teuthology.orchestra.run.trial112.stdout:Get:48 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2026-02-13T01:52:13.157 INFO:teuthology.orchestra.run.trial112.stdout:Get:49 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2026-02-13T01:52:13.158 INFO:teuthology.orchestra.run.trial112.stdout:Get:50 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-pygments all 2.11.2+dfsg-2ubuntu0.1 [750 kB] 2026-02-13T01:52:13.163 INFO:teuthology.orchestra.run.trial112.stdout:Get:51 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2026-02-13T01:52:13.174 INFO:teuthology.orchestra.run.trial112.stdout:Get:52 https://1.chacra.ceph.com/r/ceph/wip-bharath17-testing-2026-02-12-0618-tentacle/3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 20.2.0-636-g3b40640f-1jammy [833 kB] 2026-02-13T01:52:13.229 INFO:teuthology.orchestra.run.trial112.stdout:Get:53 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2026-02-13T01:52:13.231 INFO:teuthology.orchestra.run.trial112.stdout:Get:54 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2026-02-13T01:52:13.231 INFO:teuthology.orchestra.run.trial112.stdout:Get:55 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2026-02-13T01:52:13.232 INFO:teuthology.orchestra.run.trial112.stdout:Get:56 https://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1983 kB] 2026-02-13T01:52:13.366 INFO:teuthology.orchestra.run.trial112.stdout:Get:57 https://1.chacra.ceph.com/r/ceph/wip-bharath17-testing-2026-02-12-0618-tentacle/3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 20.2.0-636-g3b40640f-1jammy [364 kB] 2026-02-13T01:52:13.373 INFO:teuthology.orchestra.run.trial112.stdout:Get:58 https://1.chacra.ceph.com/r/ceph/wip-bharath17-testing-2026-02-12-0618-tentacle/3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 20.2.0-636-g3b40640f-1jammy [32.8 kB] 2026-02-13T01:52:13.373 INFO:teuthology.orchestra.run.trial112.stdout:Get:59 https://1.chacra.ceph.com/r/ceph/wip-bharath17-testing-2026-02-12-0618-tentacle/3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 20.2.0-636-g3b40640f-1jammy [184 kB] 2026-02-13T01:52:13.436 INFO:teuthology.orchestra.run.trial112.stdout:Get:60 https://1.chacra.ceph.com/r/ceph/wip-bharath17-testing-2026-02-12-0618-tentacle/3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 20.2.0-636-g3b40640f-1jammy [83.9 kB] 2026-02-13T01:52:13.437 INFO:teuthology.orchestra.run.trial112.stdout:Get:61 https://1.chacra.ceph.com/r/ceph/wip-bharath17-testing-2026-02-12-0618-tentacle/3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 20.2.0-636-g3b40640f-1jammy [340 kB] 2026-02-13T01:52:13.503 INFO:teuthology.orchestra.run.trial112.stdout:Get:62 https://1.chacra.ceph.com/r/ceph/wip-bharath17-testing-2026-02-12-0618-tentacle/3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 20.2.0-636-g3b40640f-1jammy [8687 kB] 2026-02-13T01:52:14.875 INFO:teuthology.orchestra.run.trial112.stdout:Get:63 https://1.chacra.ceph.com/r/ceph/wip-bharath17-testing-2026-02-12-0618-tentacle/3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 20.2.0-636-g3b40640f-1jammy [112 kB] 2026-02-13T01:52:14.875 INFO:teuthology.orchestra.run.trial112.stdout:Get:64 https://1.chacra.ceph.com/r/ceph/wip-bharath17-testing-2026-02-12-0618-tentacle/3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 20.2.0-636-g3b40640f-1jammy [261 kB] 2026-02-13T01:52:14.942 INFO:teuthology.orchestra.run.trial112.stdout:Get:65 https://1.chacra.ceph.com/r/ceph/wip-bharath17-testing-2026-02-12-0618-tentacle/3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 20.2.0-636-g3b40640f-1jammy [29.3 MB] 2026-02-13T01:52:24.056 INFO:teuthology.orchestra.run.trial112.stdout:Get:66 https://1.chacra.ceph.com/r/ceph/wip-bharath17-testing-2026-02-12-0618-tentacle/3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 20.2.0-636-g3b40640f-1jammy [5415 kB] 2026-02-13T01:52:25.241 INFO:teuthology.orchestra.run.trial112.stdout:Get:67 https://1.chacra.ceph.com/r/ceph/wip-bharath17-testing-2026-02-12-0618-tentacle/3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 20.2.0-636-g3b40640f-1jammy [246 kB] 2026-02-13T01:52:25.299 INFO:teuthology.orchestra.run.trial112.stdout:Get:68 https://1.chacra.ceph.com/r/ceph/wip-bharath17-testing-2026-02-12-0618-tentacle/3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 20.2.0-636-g3b40640f-1jammy [124 kB] 2026-02-13T01:52:25.311 INFO:teuthology.orchestra.run.trial112.stdout:Get:69 https://1.chacra.ceph.com/r/ceph/wip-bharath17-testing-2026-02-12-0618-tentacle/3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 20.2.0-636-g3b40640f-1jammy [906 kB] 2026-02-13T01:52:25.500 INFO:teuthology.orchestra.run.trial112.stdout:Get:70 https://1.chacra.ceph.com/r/ceph/wip-bharath17-testing-2026-02-12-0618-tentacle/3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 20.2.0-636-g3b40640f-1jammy [6399 kB] 2026-02-13T01:52:26.683 INFO:teuthology.orchestra.run.trial112.stdout:Get:71 https://1.chacra.ceph.com/r/ceph/wip-bharath17-testing-2026-02-12-0618-tentacle/3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 20.2.0-636-g3b40640f-1jammy [21.7 MB] 2026-02-13T01:52:30.901 INFO:teuthology.orchestra.run.trial112.stdout:Get:72 https://1.chacra.ceph.com/r/ceph/wip-bharath17-testing-2026-02-12-0618-tentacle/3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 20.2.0-636-g3b40640f-1jammy [14.1 kB] 2026-02-13T01:52:30.903 INFO:teuthology.orchestra.run.trial112.stdout:Get:73 https://1.chacra.ceph.com/r/ceph/wip-bharath17-testing-2026-02-12-0618-tentacle/3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 20.2.0-636-g3b40640f-1jammy [957 kB] 2026-02-13T01:52:31.091 INFO:teuthology.orchestra.run.trial112.stdout:Get:74 https://1.chacra.ceph.com/r/ceph/wip-bharath17-testing-2026-02-12-0618-tentacle/3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 20.2.0-636-g3b40640f-1jammy [2345 kB] 2026-02-13T01:52:31.559 INFO:teuthology.orchestra.run.trial112.stdout:Get:75 https://1.chacra.ceph.com/r/ceph/wip-bharath17-testing-2026-02-12-0618-tentacle/3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 20.2.0-636-g3b40640f-1jammy [1049 kB] 2026-02-13T01:52:31.764 INFO:teuthology.orchestra.run.trial112.stdout:Get:76 https://1.chacra.ceph.com/r/ceph/wip-bharath17-testing-2026-02-12-0618-tentacle/3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 20.2.0-636-g3b40640f-1jammy [179 kB] 2026-02-13T01:52:31.785 INFO:teuthology.orchestra.run.trial112.stdout:Get:77 https://1.chacra.ceph.com/r/ceph/wip-bharath17-testing-2026-02-12-0618-tentacle/3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 20.2.0-636-g3b40640f-1jammy [45.5 MB] 2026-02-13T01:52:39.179 INFO:teuthology.orchestra.run.trial112.stdout:Get:78 https://1.chacra.ceph.com/r/ceph/wip-bharath17-testing-2026-02-12-0618-tentacle/3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 20.2.0-636-g3b40640f-1jammy [8625 kB] 2026-02-13T01:52:40.819 INFO:teuthology.orchestra.run.trial112.stdout:Get:79 https://1.chacra.ceph.com/r/ceph/wip-bharath17-testing-2026-02-12-0618-tentacle/3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 20.2.0-636-g3b40640f-1jammy [14.2 kB] 2026-02-13T01:52:40.819 INFO:teuthology.orchestra.run.trial112.stdout:Get:80 https://1.chacra.ceph.com/r/ceph/wip-bharath17-testing-2026-02-12-0618-tentacle/3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 20.2.0-636-g3b40640f-1jammy [99.4 MB] 2026-02-13T01:53:14.487 INFO:teuthology.orchestra.run.trial112.stdout:Get:81 https://1.chacra.ceph.com/r/ceph/wip-bharath17-testing-2026-02-12-0618-tentacle/3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 20.2.0-636-g3b40640f-1jammy [135 kB] 2026-02-13T01:53:14.518 INFO:teuthology.orchestra.run.trial112.stdout:Get:82 https://1.chacra.ceph.com/r/ceph/wip-bharath17-testing-2026-02-12-0618-tentacle/3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-daemon amd64 20.2.0-636-g3b40640f-1jammy [43.2 kB] 2026-02-13T01:53:14.546 INFO:teuthology.orchestra.run.trial112.stdout:Get:83 https://1.chacra.ceph.com/r/ceph/wip-bharath17-testing-2026-02-12-0618-tentacle/3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-proxy2 amd64 20.2.0-636-g3b40640f-1jammy [30.7 kB] 2026-02-13T01:53:14.548 INFO:teuthology.orchestra.run.trial112.stdout:Get:84 https://1.chacra.ceph.com/r/ceph/wip-bharath17-testing-2026-02-12-0618-tentacle/3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 20.2.0-636-g3b40640f-1jammy [41.5 kB] 2026-02-13T01:53:14.554 INFO:teuthology.orchestra.run.trial112.stdout:Get:85 https://1.chacra.ceph.com/r/ceph/wip-bharath17-testing-2026-02-12-0618-tentacle/3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 20.2.0-636-g3b40640f-1jammy [25.1 MB] 2026-02-13T01:53:25.251 INFO:teuthology.orchestra.run.trial112.stdout:Get:86 https://1.chacra.ceph.com/r/ceph/wip-bharath17-testing-2026-02-12-0618-tentacle/3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 20.2.0-636-g3b40640f-1jammy [97.5 kB] 2026-02-13T01:53:25.597 INFO:teuthology.orchestra.run.trial112.stdout:Fetched 280 MB in 1min 14s (3802 kB/s) 2026-02-13T01:53:25.614 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2026-02-13T01:53:25.773 INFO:teuthology.orchestra.run.trial112.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 167438 files and directories currently installed.) 2026-02-13T01:53:25.775 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../00-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2026-02-13T01:53:25.776 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-13T01:53:25.798 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2026-02-13T01:53:25.811 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../01-libdouble-conversion3_3.1.7-4_amd64.deb ... 2026-02-13T01:53:25.811 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-13T01:53:25.827 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2026-02-13T01:53:25.839 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../02-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2026-02-13T01:53:25.840 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-13T01:53:25.859 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package libqt5core5a:amd64. 2026-02-13T01:53:25.867 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../03-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-13T01:53:25.873 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-13T01:53:25.906 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2026-02-13T01:53:25.914 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../04-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-13T01:53:25.915 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-13T01:53:25.931 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package libqt5network5:amd64. 2026-02-13T01:53:25.939 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../05-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-13T01:53:25.939 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-13T01:53:25.959 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2026-02-13T01:53:25.967 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../06-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2026-02-13T01:53:25.968 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-13T01:53:25.984 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package librados2. 2026-02-13T01:53:25.992 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../07-librados2_20.2.0-636-g3b40640f-1jammy_amd64.deb ... 2026-02-13T01:53:25.993 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking librados2 (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:26.034 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package libnbd0. 2026-02-13T01:53:26.043 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../08-libnbd0_1.10.5-1_amd64.deb ... 2026-02-13T01:53:26.043 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking libnbd0 (1.10.5-1) ... 2026-02-13T01:53:26.058 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package librbd1. 2026-02-13T01:53:26.066 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../09-librbd1_20.2.0-636-g3b40640f-1jammy_amd64.deb ... 2026-02-13T01:53:26.066 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking librbd1 (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:26.103 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package libcephfs2. 2026-02-13T01:53:26.112 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../10-libcephfs2_20.2.0-636-g3b40640f-1jammy_amd64.deb ... 2026-02-13T01:53:26.112 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking libcephfs2 (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:26.132 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python3-rados. 2026-02-13T01:53:26.140 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../11-python3-rados_20.2.0-636-g3b40640f-1jammy_amd64.deb ... 2026-02-13T01:53:26.141 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python3-rados (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:26.158 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python3-ceph-argparse. 2026-02-13T01:53:26.166 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../12-python3-ceph-argparse_20.2.0-636-g3b40640f-1jammy_all.deb ... 2026-02-13T01:53:26.167 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python3-ceph-argparse (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:26.180 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python3-cephfs. 2026-02-13T01:53:26.188 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../13-python3-cephfs_20.2.0-636-g3b40640f-1jammy_amd64.deb ... 2026-02-13T01:53:26.189 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python3-cephfs (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:26.204 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python3-ceph-common. 2026-02-13T01:53:26.213 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../14-python3-ceph-common_20.2.0-636-g3b40640f-1jammy_all.deb ... 2026-02-13T01:53:26.213 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python3-ceph-common (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:26.230 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python3-wcwidth. 2026-02-13T01:53:26.238 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../15-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2026-02-13T01:53:26.239 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-13T01:53:26.255 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python3-prettytable. 2026-02-13T01:53:26.263 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../16-python3-prettytable_2.5.0-2_all.deb ... 2026-02-13T01:53:26.264 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2026-02-13T01:53:26.278 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python3-rbd. 2026-02-13T01:53:26.286 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../17-python3-rbd_20.2.0-636-g3b40640f-1jammy_amd64.deb ... 2026-02-13T01:53:26.287 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python3-rbd (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:26.304 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package librdkafka1:amd64. 2026-02-13T01:53:26.312 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../18-librdkafka1_1.8.0-1build1_amd64.deb ... 2026-02-13T01:53:26.313 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-13T01:53:26.331 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package librgw2. 2026-02-13T01:53:26.339 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../19-librgw2_20.2.0-636-g3b40640f-1jammy_amd64.deb ... 2026-02-13T01:53:26.339 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking librgw2 (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:26.437 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python3-rgw. 2026-02-13T01:53:26.445 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../20-python3-rgw_20.2.0-636-g3b40640f-1jammy_amd64.deb ... 2026-02-13T01:53:26.446 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python3-rgw (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:26.460 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package liboath0:amd64. 2026-02-13T01:53:26.469 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../21-liboath0_2.6.7-3ubuntu0.1_amd64.deb ... 2026-02-13T01:53:26.469 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-13T01:53:26.483 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package libradosstriper1. 2026-02-13T01:53:26.492 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../22-libradosstriper1_20.2.0-636-g3b40640f-1jammy_amd64.deb ... 2026-02-13T01:53:26.492 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking libradosstriper1 (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:26.508 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package ceph-common. 2026-02-13T01:53:26.516 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../23-ceph-common_20.2.0-636-g3b40640f-1jammy_amd64.deb ... 2026-02-13T01:53:26.516 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking ceph-common (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:26.768 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package ceph-base. 2026-02-13T01:53:26.777 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../24-ceph-base_20.2.0-636-g3b40640f-1jammy_amd64.deb ... 2026-02-13T01:53:26.782 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking ceph-base (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:26.846 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python3-jaraco.functools. 2026-02-13T01:53:26.855 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../25-python3-jaraco.functools_3.4.0-2_all.deb ... 2026-02-13T01:53:26.855 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2026-02-13T01:53:26.870 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python3-cheroot. 2026-02-13T01:53:26.879 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../26-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2026-02-13T01:53:26.879 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-13T01:53:26.896 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python3-jaraco.classes. 2026-02-13T01:53:26.904 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../27-python3-jaraco.classes_3.2.1-3_all.deb ... 2026-02-13T01:53:26.905 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2026-02-13T01:53:26.919 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python3-jaraco.text. 2026-02-13T01:53:26.927 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../28-python3-jaraco.text_3.6.0-2_all.deb ... 2026-02-13T01:53:26.928 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2026-02-13T01:53:26.942 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python3-jaraco.collections. 2026-02-13T01:53:26.951 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../29-python3-jaraco.collections_3.4.0-2_all.deb ... 2026-02-13T01:53:26.951 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2026-02-13T01:53:26.965 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python3-tempora. 2026-02-13T01:53:26.974 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../30-python3-tempora_4.1.2-1_all.deb ... 2026-02-13T01:53:26.974 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python3-tempora (4.1.2-1) ... 2026-02-13T01:53:26.989 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python3-portend. 2026-02-13T01:53:26.998 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../31-python3-portend_3.0.0-1_all.deb ... 2026-02-13T01:53:26.998 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python3-portend (3.0.0-1) ... 2026-02-13T01:53:27.011 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python3-zc.lockfile. 2026-02-13T01:53:27.020 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../32-python3-zc.lockfile_2.0-1_all.deb ... 2026-02-13T01:53:27.020 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2026-02-13T01:53:27.034 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python3-cherrypy3. 2026-02-13T01:53:27.043 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../33-python3-cherrypy3_18.6.1-4_all.deb ... 2026-02-13T01:53:27.043 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2026-02-13T01:53:27.067 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python3-natsort. 2026-02-13T01:53:27.075 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../34-python3-natsort_8.0.2-1_all.deb ... 2026-02-13T01:53:27.076 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python3-natsort (8.0.2-1) ... 2026-02-13T01:53:27.091 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2026-02-13T01:53:27.099 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../35-ceph-mgr-modules-core_20.2.0-636-g3b40640f-1jammy_all.deb ... 2026-02-13T01:53:27.100 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking ceph-mgr-modules-core (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:27.126 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2026-02-13T01:53:27.134 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../36-libsqlite3-mod-ceph_20.2.0-636-g3b40640f-1jammy_amd64.deb ... 2026-02-13T01:53:27.135 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking libsqlite3-mod-ceph (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:27.149 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package ceph-mgr. 2026-02-13T01:53:27.157 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../37-ceph-mgr_20.2.0-636-g3b40640f-1jammy_amd64.deb ... 2026-02-13T01:53:27.157 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking ceph-mgr (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:27.178 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package ceph-mon. 2026-02-13T01:53:27.186 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../38-ceph-mon_20.2.0-636-g3b40640f-1jammy_amd64.deb ... 2026-02-13T01:53:27.187 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking ceph-mon (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:27.246 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package libfuse2:amd64. 2026-02-13T01:53:27.255 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../39-libfuse2_2.9.9-5ubuntu3_amd64.deb ... 2026-02-13T01:53:27.255 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-13T01:53:27.271 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package ceph-osd. 2026-02-13T01:53:27.279 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../40-ceph-osd_20.2.0-636-g3b40640f-1jammy_amd64.deb ... 2026-02-13T01:53:27.279 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking ceph-osd (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:27.424 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package ceph. 2026-02-13T01:53:27.432 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../41-ceph_20.2.0-636-g3b40640f-1jammy_amd64.deb ... 2026-02-13T01:53:27.433 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking ceph (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:27.447 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package ceph-fuse. 2026-02-13T01:53:27.455 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../42-ceph-fuse_20.2.0-636-g3b40640f-1jammy_amd64.deb ... 2026-02-13T01:53:27.455 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking ceph-fuse (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:27.476 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package ceph-mds. 2026-02-13T01:53:27.484 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../43-ceph-mds_20.2.0-636-g3b40640f-1jammy_amd64.deb ... 2026-02-13T01:53:27.485 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking ceph-mds (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:27.516 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package cephadm. 2026-02-13T01:53:27.525 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../44-cephadm_20.2.0-636-g3b40640f-1jammy_amd64.deb ... 2026-02-13T01:53:27.525 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking cephadm (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:27.541 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python3-asyncssh. 2026-02-13T01:53:27.550 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../45-python3-asyncssh_2.5.0-1ubuntu0.1_all.deb ... 2026-02-13T01:53:27.550 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-13T01:53:27.572 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2026-02-13T01:53:27.580 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../46-ceph-mgr-cephadm_20.2.0-636-g3b40640f-1jammy_all.deb ... 2026-02-13T01:53:27.581 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking ceph-mgr-cephadm (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:27.602 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python3-repoze.lru. 2026-02-13T01:53:27.610 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../47-python3-repoze.lru_0.7-2_all.deb ... 2026-02-13T01:53:27.610 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2026-02-13T01:53:27.626 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python3-routes. 2026-02-13T01:53:27.635 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../48-python3-routes_2.5.1-1ubuntu1_all.deb ... 2026-02-13T01:53:27.635 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2026-02-13T01:53:27.655 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2026-02-13T01:53:27.663 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../49-ceph-mgr-dashboard_20.2.0-636-g3b40640f-1jammy_all.deb ... 2026-02-13T01:53:27.664 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking ceph-mgr-dashboard (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:28.087 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2026-02-13T01:53:28.096 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../50-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2026-02-13T01:53:28.096 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-13T01:53:28.134 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python3-joblib. 2026-02-13T01:53:28.143 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../51-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2026-02-13T01:53:28.143 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2026-02-13T01:53:28.170 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python3-threadpoolctl. 2026-02-13T01:53:28.178 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../52-python3-threadpoolctl_3.1.0-1_all.deb ... 2026-02-13T01:53:28.179 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2026-02-13T01:53:28.194 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python3-sklearn. 2026-02-13T01:53:28.201 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../53-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2026-02-13T01:53:28.202 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-13T01:53:28.283 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2026-02-13T01:53:28.292 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../54-ceph-mgr-diskprediction-local_20.2.0-636-g3b40640f-1jammy_all.deb ... 2026-02-13T01:53:28.292 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking ceph-mgr-diskprediction-local (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:28.459 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python3-cachetools. 2026-02-13T01:53:28.467 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../55-python3-cachetools_5.0.0-1_all.deb ... 2026-02-13T01:53:28.468 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2026-02-13T01:53:28.482 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python3-rsa. 2026-02-13T01:53:28.491 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../56-python3-rsa_4.8-1_all.deb ... 2026-02-13T01:53:28.492 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python3-rsa (4.8-1) ... 2026-02-13T01:53:28.509 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python3-google-auth. 2026-02-13T01:53:28.518 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../57-python3-google-auth_1.5.1-3_all.deb ... 2026-02-13T01:53:28.519 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2026-02-13T01:53:28.535 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python3-requests-oauthlib. 2026-02-13T01:53:28.544 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../58-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2026-02-13T01:53:28.544 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-13T01:53:28.560 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python3-websocket. 2026-02-13T01:53:28.568 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../59-python3-websocket_1.2.3-1_all.deb ... 2026-02-13T01:53:28.569 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python3-websocket (1.2.3-1) ... 2026-02-13T01:53:28.586 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python3-kubernetes. 2026-02-13T01:53:28.594 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../60-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2026-02-13T01:53:28.595 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-13T01:53:28.690 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2026-02-13T01:53:28.699 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../61-ceph-mgr-k8sevents_20.2.0-636-g3b40640f-1jammy_all.deb ... 2026-02-13T01:53:28.699 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking ceph-mgr-k8sevents (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:28.714 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package libonig5:amd64. 2026-02-13T01:53:28.722 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../62-libonig5_6.9.7.1-2build1_amd64.deb ... 2026-02-13T01:53:28.723 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-13T01:53:28.738 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package libjq1:amd64. 2026-02-13T01:53:28.747 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../63-libjq1_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-13T01:53:28.747 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-13T01:53:28.762 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package jq. 2026-02-13T01:53:28.770 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../64-jq_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-13T01:53:28.770 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking jq (1.6-2.1ubuntu3.1) ... 2026-02-13T01:53:28.784 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package socat. 2026-02-13T01:53:28.793 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../65-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2026-02-13T01:53:28.793 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2026-02-13T01:53:28.814 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package xmlstarlet. 2026-02-13T01:53:28.822 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../66-xmlstarlet_1.6.1-2.1_amd64.deb ... 2026-02-13T01:53:28.822 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2026-02-13T01:53:28.857 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package ceph-test. 2026-02-13T01:53:28.865 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../67-ceph-test_20.2.0-636-g3b40640f-1jammy_amd64.deb ... 2026-02-13T01:53:28.866 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking ceph-test (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:29.776 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package ceph-volume. 2026-02-13T01:53:29.785 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../68-ceph-volume_20.2.0-636-g3b40640f-1jammy_all.deb ... 2026-02-13T01:53:29.785 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking ceph-volume (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:29.807 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package libcephfs-daemon. 2026-02-13T01:53:29.816 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../69-libcephfs-daemon_20.2.0-636-g3b40640f-1jammy_amd64.deb ... 2026-02-13T01:53:29.816 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking libcephfs-daemon (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:29.831 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package libcephfs-proxy2. 2026-02-13T01:53:29.840 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../70-libcephfs-proxy2_20.2.0-636-g3b40640f-1jammy_amd64.deb ... 2026-02-13T01:53:29.840 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking libcephfs-proxy2 (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:29.854 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package libcephfs-dev. 2026-02-13T01:53:29.863 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../71-libcephfs-dev_20.2.0-636-g3b40640f-1jammy_amd64.deb ... 2026-02-13T01:53:29.863 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking libcephfs-dev (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:29.879 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package nvme-cli. 2026-02-13T01:53:29.887 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../72-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2026-02-13T01:53:29.888 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2026-02-13T01:53:29.918 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python-asyncssh-doc. 2026-02-13T01:53:29.926 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../73-python-asyncssh-doc_2.5.0-1ubuntu0.1_all.deb ... 2026-02-13T01:53:29.927 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-13T01:53:29.956 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python3-iniconfig. 2026-02-13T01:53:29.965 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../74-python3-iniconfig_1.1.1-2_all.deb ... 2026-02-13T01:53:29.965 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2026-02-13T01:53:29.980 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python3-pluggy. 2026-02-13T01:53:29.989 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../75-python3-pluggy_0.13.0-7.1_all.deb ... 2026-02-13T01:53:29.989 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2026-02-13T01:53:30.005 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python3-psutil. 2026-02-13T01:53:30.014 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../76-python3-psutil_5.9.0-1build1_amd64.deb ... 2026-02-13T01:53:30.014 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2026-02-13T01:53:30.032 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python3-py. 2026-02-13T01:53:30.041 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../77-python3-py_1.10.0-1_all.deb ... 2026-02-13T01:53:30.041 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python3-py (1.10.0-1) ... 2026-02-13T01:53:30.061 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python3-pygments. 2026-02-13T01:53:30.070 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../78-python3-pygments_2.11.2+dfsg-2ubuntu0.1_all.deb ... 2026-02-13T01:53:30.070 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-13T01:53:30.111 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python3-toml. 2026-02-13T01:53:30.120 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../79-python3-toml_0.10.2-1_all.deb ... 2026-02-13T01:53:30.120 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python3-toml (0.10.2-1) ... 2026-02-13T01:53:30.135 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python3-pytest. 2026-02-13T01:53:30.143 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../80-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2026-02-13T01:53:30.144 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2026-02-13T01:53:30.166 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python3-simplejson. 2026-02-13T01:53:30.175 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../81-python3-simplejson_3.17.6-1build1_amd64.deb ... 2026-02-13T01:53:30.175 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2026-02-13T01:53:30.193 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package python3-webob. 2026-02-13T01:53:30.201 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../82-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2026-02-13T01:53:30.202 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-13T01:53:30.225 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package qttranslations5-l10n. 2026-02-13T01:53:30.233 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../83-qttranslations5-l10n_5.15.3-1_all.deb ... 2026-02-13T01:53:30.234 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2026-02-13T01:53:30.301 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package radosgw. 2026-02-13T01:53:30.310 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../84-radosgw_20.2.0-636-g3b40640f-1jammy_amd64.deb ... 2026-02-13T01:53:30.311 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking radosgw (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:30.524 INFO:teuthology.orchestra.run.trial112.stdout:Selecting previously unselected package rbd-fuse. 2026-02-13T01:53:30.533 INFO:teuthology.orchestra.run.trial112.stdout:Preparing to unpack .../85-rbd-fuse_20.2.0-636-g3b40640f-1jammy_amd64.deb ... 2026-02-13T01:53:30.534 INFO:teuthology.orchestra.run.trial112.stdout:Unpacking rbd-fuse (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:30.561 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2026-02-13T01:53:30.629 INFO:teuthology.orchestra.run.trial112.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-13T01:53:30.630 INFO:teuthology.orchestra.run.trial112.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2026-02-13T01:53:30.702 INFO:teuthology.orchestra.run.trial112.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /lib/systemd/system/nvmefc-boot-connections.service. 2026-02-13T01:53:30.986 INFO:teuthology.orchestra.run.trial112.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service → /lib/systemd/system/nvmf-autoconnect.service. 2026-02-13T01:53:31.351 INFO:teuthology.orchestra.run.trial112.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2026-02-13T01:53:31.362 INFO:teuthology.orchestra.run.trial112.stdout:Setting up cephadm (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:31.404 INFO:teuthology.orchestra.run.trial112.stdout:Adding system user cephadm....done 2026-02-13T01:53:31.414 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2026-02-13T01:53:31.486 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-13T01:53:31.487 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2026-02-13T01:53:31.558 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python3-repoze.lru (0.7-2) ... 2026-02-13T01:53:31.633 INFO:teuthology.orchestra.run.trial112.stdout:Setting up liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-13T01:53:31.634 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python3-py (1.10.0-1) ... 2026-02-13T01:53:31.731 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2026-02-13T01:53:31.853 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python3-cachetools (5.0.0-1) ... 2026-02-13T01:53:31.928 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2026-02-13T01:53:32.004 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python3-ceph-argparse (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:32.071 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-13T01:53:32.072 INFO:teuthology.orchestra.run.trial112.stdout:Setting up libnbd0 (1.10.5-1) ... 2026-02-13T01:53:32.073 INFO:teuthology.orchestra.run.trial112.stdout:Setting up libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-13T01:53:32.074 INFO:teuthology.orchestra.run.trial112.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-13T01:53:32.075 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2026-02-13T01:53:32.196 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python3-natsort (8.0.2-1) ... 2026-02-13T01:53:32.276 INFO:teuthology.orchestra.run.trial112.stdout:Setting up libcephfs-proxy2 (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:32.277 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2026-02-13T01:53:32.355 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2026-02-13T01:53:32.443 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-13T01:53:32.700 INFO:teuthology.orchestra.run.trial112.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2026-02-13T01:53:32.701 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-13T01:53:32.796 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-13T01:53:32.922 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-13T01:53:33.003 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2026-02-13T01:53:33.074 INFO:teuthology.orchestra.run.trial112.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2026-02-13T01:53:33.075 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python3-ceph-common (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:33.163 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-13T01:53:33.670 INFO:teuthology.orchestra.run.trial112.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-13T01:53:33.674 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python3-toml (0.10.2-1) ... 2026-02-13T01:53:33.751 INFO:teuthology.orchestra.run.trial112.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-13T01:53:33.752 INFO:teuthology.orchestra.run.trial112.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2026-02-13T01:53:33.753 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2026-02-13T01:53:33.829 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2026-02-13T01:53:33.904 INFO:teuthology.orchestra.run.trial112.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-13T01:53:33.905 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python3-rsa (4.8-1) ... 2026-02-13T01:53:33.983 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python3-tempora (4.1.2-1) ... 2026-02-13T01:53:34.055 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python3-prettytable (2.5.0-2) ... 2026-02-13T01:53:34.131 INFO:teuthology.orchestra.run.trial112.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-13T01:53:34.132 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python3-websocket (1.2.3-1) ... 2026-02-13T01:53:34.213 INFO:teuthology.orchestra.run.trial112.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-13T01:53:34.214 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-13T01:53:34.290 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-13T01:53:34.375 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2026-02-13T01:53:34.447 INFO:teuthology.orchestra.run.trial112.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-13T01:53:34.448 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2026-02-13T01:53:34.582 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python3-portend (3.0.0-1) ... 2026-02-13T01:53:34.654 INFO:teuthology.orchestra.run.trial112.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-13T01:53:34.655 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python3-google-auth (1.5.1-3) ... 2026-02-13T01:53:34.728 INFO:teuthology.orchestra.run.trial112.stdout:Setting up jq (1.6-2.1ubuntu3.1) ... 2026-02-13T01:53:34.729 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2026-02-13T01:53:34.861 INFO:teuthology.orchestra.run.trial112.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-13T01:53:34.862 INFO:teuthology.orchestra.run.trial112.stdout:Setting up librados2 (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:34.863 INFO:teuthology.orchestra.run.trial112.stdout:Setting up librgw2 (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:34.864 INFO:teuthology.orchestra.run.trial112.stdout:Setting up libsqlite3-mod-ceph (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:34.865 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-13T01:53:35.387 INFO:teuthology.orchestra.run.trial112.stdout:Setting up libcephfs2 (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:35.388 INFO:teuthology.orchestra.run.trial112.stdout:Setting up libradosstriper1 (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:35.389 INFO:teuthology.orchestra.run.trial112.stdout:Setting up librbd1 (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:35.390 INFO:teuthology.orchestra.run.trial112.stdout:Setting up ceph-mgr-modules-core (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:35.391 INFO:teuthology.orchestra.run.trial112.stdout:Setting up ceph-fuse (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:35.461 INFO:teuthology.orchestra.run.trial112.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-13T01:53:35.462 INFO:teuthology.orchestra.run.trial112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-13T01:53:35.836 INFO:teuthology.orchestra.run.trial112.stdout:Setting up libcephfs-dev (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:35.837 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python3-rados (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:35.838 INFO:teuthology.orchestra.run.trial112.stdout:Setting up libcephfs-daemon (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:35.838 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python3-rbd (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:35.839 INFO:teuthology.orchestra.run.trial112.stdout:Setting up rbd-fuse (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:35.840 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python3-rgw (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:35.841 INFO:teuthology.orchestra.run.trial112.stdout:Setting up python3-cephfs (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:35.842 INFO:teuthology.orchestra.run.trial112.stdout:Setting up ceph-common (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:35.870 INFO:teuthology.orchestra.run.trial112.stdout:Adding group ceph....done 2026-02-13T01:53:35.905 INFO:teuthology.orchestra.run.trial112.stdout:Adding system user ceph....done 2026-02-13T01:53:35.915 INFO:teuthology.orchestra.run.trial112.stdout:Setting system user ceph properties....done 2026-02-13T01:53:35.922 INFO:teuthology.orchestra.run.trial112.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2026-02-13T01:53:36.010 INFO:teuthology.orchestra.run.trial112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /lib/systemd/system/ceph.target. 2026-02-13T01:53:36.298 INFO:teuthology.orchestra.run.trial112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service → /lib/systemd/system/rbdmap.service. 2026-02-13T01:53:36.646 INFO:teuthology.orchestra.run.trial112.stdout:Setting up ceph-test (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:36.647 INFO:teuthology.orchestra.run.trial112.stdout:Setting up radosgw (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:36.890 INFO:teuthology.orchestra.run.trial112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-13T01:53:36.890 INFO:teuthology.orchestra.run.trial112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-13T01:53:37.238 INFO:teuthology.orchestra.run.trial112.stdout:Setting up ceph-base (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:37.356 INFO:teuthology.orchestra.run.trial112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /lib/systemd/system/ceph-crash.service. 2026-02-13T01:53:37.706 INFO:teuthology.orchestra.run.trial112.stdout:Setting up ceph-mds (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:37.767 INFO:teuthology.orchestra.run.trial112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-13T01:53:37.767 INFO:teuthology.orchestra.run.trial112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-13T01:53:38.124 INFO:teuthology.orchestra.run.trial112.stdout:Setting up ceph-mgr (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:38.204 INFO:teuthology.orchestra.run.trial112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-13T01:53:38.204 INFO:teuthology.orchestra.run.trial112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-13T01:53:38.558 INFO:teuthology.orchestra.run.trial112.stdout:Setting up ceph-osd (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:38.646 INFO:teuthology.orchestra.run.trial112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-13T01:53:38.646 INFO:teuthology.orchestra.run.trial112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-13T01:53:38.983 INFO:teuthology.orchestra.run.trial112.stdout:Setting up ceph-mgr-k8sevents (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:38.992 INFO:teuthology.orchestra.run.trial112.stdout:Setting up ceph-mgr-diskprediction-local (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:39.004 INFO:teuthology.orchestra.run.trial112.stdout:Setting up ceph-mon (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:39.078 INFO:teuthology.orchestra.run.trial112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-13T01:53:39.078 INFO:teuthology.orchestra.run.trial112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-13T01:53:39.446 INFO:teuthology.orchestra.run.trial112.stdout:Setting up ceph-mgr-cephadm (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:39.457 INFO:teuthology.orchestra.run.trial112.stdout:Setting up ceph (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:39.458 INFO:teuthology.orchestra.run.trial112.stdout:Setting up ceph-mgr-dashboard (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:39.469 INFO:teuthology.orchestra.run.trial112.stdout:Setting up ceph-volume (20.2.0-636-g3b40640f-1jammy) ... 2026-02-13T01:53:39.582 INFO:teuthology.orchestra.run.trial112.stdout:Processing triggers for man-db (2.10.2-1) ... 2026-02-13T01:53:39.668 INFO:teuthology.orchestra.run.trial112.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.11) ... 2026-02-13T01:53:40.101 INFO:teuthology.orchestra.run.trial112.stdout: 2026-02-13T01:53:40.101 INFO:teuthology.orchestra.run.trial112.stdout:Running kernel seems to be up-to-date. 2026-02-13T01:53:40.101 INFO:teuthology.orchestra.run.trial112.stdout: 2026-02-13T01:53:40.101 INFO:teuthology.orchestra.run.trial112.stdout:The processor microcode seems to be up-to-date. 2026-02-13T01:53:40.101 INFO:teuthology.orchestra.run.trial112.stdout: 2026-02-13T01:53:40.102 INFO:teuthology.orchestra.run.trial112.stdout:No services need to be restarted. 2026-02-13T01:53:40.102 INFO:teuthology.orchestra.run.trial112.stdout: 2026-02-13T01:53:40.102 INFO:teuthology.orchestra.run.trial112.stdout:No containers need to be restarted. 2026-02-13T01:53:40.102 INFO:teuthology.orchestra.run.trial112.stdout: 2026-02-13T01:53:40.102 INFO:teuthology.orchestra.run.trial112.stdout:No user sessions are running outdated binaries. 2026-02-13T01:53:40.102 INFO:teuthology.orchestra.run.trial112.stdout: 2026-02-13T01:53:40.102 INFO:teuthology.orchestra.run.trial112.stdout:No VM guests are running outdated hypervisor (qemu) binaries on this host. 2026-02-13T01:53:40.741 INFO:teuthology.orchestra.run.trial112.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2026-02-13T01:53:40.748 DEBUG:teuthology.parallel:result is None 2026-02-13T01:53:40.748 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb 2026-02-13T01:53:40.913 DEBUG:teuthology.orchestra.run.trial112:> dpkg-query -W -f '${Version}' ceph 2026-02-13T01:53:40.924 INFO:teuthology.orchestra.run.trial112.stdout:20.2.0-636-g3b40640f-1jammy 2026-02-13T01:53:40.925 INFO:teuthology.packaging:The installed version of ceph is 20.2.0-636-g3b40640f-1jammy 2026-02-13T01:53:40.925 INFO:teuthology.task.install:The correct ceph version 20.2.0-636-g3b40640f-1jammy is installed. 2026-02-13T01:53:40.927 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-13T01:53:40.927 DEBUG:teuthology.orchestra.run.trial112:> set -ex 2026-02-13T01:53:40.928 DEBUG:teuthology.orchestra.run.trial112:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-13T01:53:40.973 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-13T01:53:40.974 DEBUG:teuthology.orchestra.run.trial112:> set -ex 2026-02-13T01:53:40.974 DEBUG:teuthology.orchestra.run.trial112:> sudo dd of=/usr/bin/daemon-helper 2026-02-13T01:53:41.024 DEBUG:teuthology.orchestra.run.trial112:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-13T01:53:41.072 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-13T01:53:41.073 DEBUG:teuthology.orchestra.run.trial112:> set -ex 2026-02-13T01:53:41.073 DEBUG:teuthology.orchestra.run.trial112:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-13T01:53:41.121 DEBUG:teuthology.orchestra.run.trial112:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-13T01:53:41.173 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-13T01:53:41.174 DEBUG:teuthology.orchestra.run.trial112:> set -ex 2026-02-13T01:53:41.174 DEBUG:teuthology.orchestra.run.trial112:> sudo dd of=/usr/bin/stdin-killer 2026-02-13T01:53:41.230 DEBUG:teuthology.orchestra.run.trial112:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-13T01:53:41.281 INFO:teuthology.run_tasks:Running task workunit... 2026-02-13T01:53:41.308 INFO:tasks.workunit:Pulling workunits from ref 3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb 2026-02-13T01:53:41.308 INFO:tasks.workunit:Making a separate scratch dir for every client... 2026-02-13T01:53:41.308 INFO:tasks.workunit:timeout=3h 2026-02-13T01:53:41.308 INFO:tasks.workunit:cleanup=True 2026-02-13T01:53:41.308 DEBUG:teuthology.orchestra.run.trial112:> stat -- /home/ubuntu/cephtest/mnt.0 2026-02-13T01:53:41.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-13T01:53:41.329 INFO:teuthology.orchestra.run.trial112.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2026-02-13T01:53:41.330 DEBUG:teuthology.orchestra.run.trial112:> mkdir -- /home/ubuntu/cephtest/mnt.0 2026-02-13T01:53:41.377 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2026-02-13T01:53:41.377 DEBUG:teuthology.orchestra.run.trial112:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2026-02-13T01:53:41.421 DEBUG:teuthology.orchestra.run.trial112:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb 2026-02-13T01:53:41.471 INFO:tasks.workunit.client.0.trial112.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2026-02-13T01:54:03.592 INFO:tasks.workunit.client.0.trial112.stderr:Note: switching to '3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb'. 2026-02-13T01:54:03.592 INFO:tasks.workunit.client.0.trial112.stderr: 2026-02-13T01:54:03.593 INFO:tasks.workunit.client.0.trial112.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2026-02-13T01:54:03.593 INFO:tasks.workunit.client.0.trial112.stderr:changes and commit them, and you can discard any commits you make in this 2026-02-13T01:54:03.593 INFO:tasks.workunit.client.0.trial112.stderr:state without impacting any branches by switching back to a branch. 2026-02-13T01:54:03.593 INFO:tasks.workunit.client.0.trial112.stderr: 2026-02-13T01:54:03.593 INFO:tasks.workunit.client.0.trial112.stderr:If you want to create a new branch to retain commits you create, you may 2026-02-13T01:54:03.593 INFO:tasks.workunit.client.0.trial112.stderr:do so (now or later) by using -c with the switch command. Example: 2026-02-13T01:54:03.593 INFO:tasks.workunit.client.0.trial112.stderr: 2026-02-13T01:54:03.593 INFO:tasks.workunit.client.0.trial112.stderr: git switch -c 2026-02-13T01:54:03.593 INFO:tasks.workunit.client.0.trial112.stderr: 2026-02-13T01:54:03.593 INFO:tasks.workunit.client.0.trial112.stderr:Or undo this operation with: 2026-02-13T01:54:03.594 INFO:tasks.workunit.client.0.trial112.stderr: 2026-02-13T01:54:03.594 INFO:tasks.workunit.client.0.trial112.stderr: git switch - 2026-02-13T01:54:03.594 INFO:tasks.workunit.client.0.trial112.stderr: 2026-02-13T01:54:03.594 INFO:tasks.workunit.client.0.trial112.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2026-02-13T01:54:03.594 INFO:tasks.workunit.client.0.trial112.stderr: 2026-02-13T01:54:03.594 INFO:tasks.workunit.client.0.trial112.stderr:HEAD is now at 3b40640f18f Merge branch 'wip-74888-tentacle' of https://github.com/aainscow/ceph into wip-bharath17-testing-2026-02-12-0618-tentacle 2026-02-13T01:54:03.602 DEBUG:teuthology.orchestra.run.trial112:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2026-02-13T01:54:03.651 DEBUG:teuthology.orchestra.run.trial112:> set -ex 2026-02-13T01:54:03.651 DEBUG:teuthology.orchestra.run.trial112:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2026-02-13T01:54:03.698 INFO:tasks.workunit:Running workunits matching scrub on client.0... 2026-02-13T01:54:03.699 INFO:tasks.workunit:Running workunit scrub/osd-mapper.sh... 2026-02-13T01:54:03.699 DEBUG:teuthology.orchestra.run.trial112:workunit test scrub/osd-mapper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh 2026-02-13T01:54:03.751 INFO:tasks.workunit.client.0.trial112.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-13T01:54:03.757 INFO:tasks.workunit.client.0.trial112.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-13T01:54:03.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-13T01:54:03.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-13T01:54:03.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export PYTHONWARNINGS=ignore 2026-02-13T01:54:03.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: PYTHONWARNINGS=ignore 2026-02-13T01:54:03.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: export CEPH_CONF=/dev/null 2026-02-13T01:54:03.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: CEPH_CONF=/dev/null 2026-02-13T01:54:03.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: main: unset CEPH_ARGS 2026-02-13T01:54:03.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: local code 2026-02-13T01:54:03.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: run td/osd-mapper 2026-02-13T01:54:03.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:11: run: local dir=td/osd-mapper 2026-02-13T01:54:03.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:12: run: shift 2026-02-13T01:54:03.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: export CEPH_MON=127.0.0.1:7144 2026-02-13T01:54:03.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: CEPH_MON=127.0.0.1:7144 2026-02-13T01:54:03.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:15: run: export CEPH_ARGS 2026-02-13T01:54:03.758 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: uuidgen 2026-02-13T01:54:03.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: CEPH_ARGS+='--fsid=b79cf458-1242-4cc6-8f9e-481a9aefbdea --auth-supported=none ' 2026-02-13T01:54:03.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:17: run: CEPH_ARGS+='--mon-host=127.0.0.1:7144 ' 2026-02-13T01:54:03.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:19: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2026-02-13T01:54:03.759 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: set 2026-02-13T01:54:03.759 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-13T01:54:03.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: local funcs=TEST_truncated_sna_record 2026-02-13T01:54:03.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:21: run: for func in $funcs 2026-02-13T01:54:03.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:22: run: setup td/osd-mapper 2026-02-13T01:54:03.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-mapper 2026-02-13T01:54:03.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-mapper 2026-02-13T01:54:03.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-mapper 2026-02-13T01:54:03.762 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T01:54:03.762 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-mapper KILL 2026-02-13T01:54:03.762 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T01:54:03.762 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T01:54:03.762 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T01:54:03.762 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T01:54:03.762 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T01:54:03.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T01:54:03.763 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T01:54:03.764 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T01:54:03.764 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T01:54:03.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T01:54:03.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T01:54:03.766 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T01:54:03.766 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T01:54:03.767 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T01:54:03.767 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T01:54:03.767 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T01:54:03.768 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T01:54:03.770 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T01:54:03.770 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-mapper 2026-02-13T01:54:03.771 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T01:54:03.771 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:54:03.771 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7480 2026-02-13T01:54:03.771 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.7480 2026-02-13T01:54:03.773 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T01:54:03.773 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T01:54:03.773 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-mapper 2026-02-13T01:54:03.774 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T01:54:03.774 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:54:03.775 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7480 2026-02-13T01:54:03.775 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.7480 2026-02-13T01:54:03.776 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T01:54:03.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-13T01:54:03.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-13T01:54:03.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T01:54:03.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-mapper 1' TERM HUP INT 2026-02-13T01:54:03.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:23: run: TEST_truncated_sna_record td/osd-mapper 2026-02-13T01:54:03.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:45: TEST_truncated_sna_record: local dir=td/osd-mapper 2026-02-13T01:54:03.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='4' ['pool_name']='test') 2026-02-13T01:54:03.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: local -A cluster_conf 2026-02-13T01:54:03.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:52: TEST_truncated_sna_record: local extr_dbg=3 2026-02-13T01:54:03.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: (( extr_dbg > 1 )) 2026-02-13T01:54:03.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: echo 'Dir: td/osd-mapper' 2026-02-13T01:54:03.779 INFO:tasks.workunit.client.0.trial112.stdout:Dir: td/osd-mapper 2026-02-13T01:54:03.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:54: TEST_truncated_sna_record: standard_scrub_cluster td/osd-mapper cluster_conf 2026-02-13T01:54:03.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local dir=td/osd-mapper 2026-02-13T01:54:03.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:230: standard_scrub_cluster: local -n args=cluster_conf 2026-02-13T01:54:03.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local OSDS=3 2026-02-13T01:54:03.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: local pg_num=4 2026-02-13T01:54:03.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:235: standard_scrub_cluster: local poolname=test 2026-02-13T01:54:03.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:236: standard_scrub_cluster: args['pool_name']=test 2026-02-13T01:54:03.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:238: standard_scrub_cluster: local pool_default_size=3 2026-02-13T01:54:03.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:239: standard_scrub_cluster: args['pool_default_size']=3 2026-02-13T01:54:03.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:241: standard_scrub_cluster: local extra_pars= 2026-02-13T01:54:03.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:242: standard_scrub_cluster: local debug_msg=dbg 2026-02-13T01:54:03.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:245: standard_scrub_cluster: local saved_echo_flag=x 2026-02-13T01:54:03.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:246: standard_scrub_cluster: set +x 2026-02-13T01:54:04.089 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 fa047f72-1d01-456d-bffe-31a04cd90f71 2026-02-13T01:54:04.192 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T01:54:04.219 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:04.213+0000 7f397948fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:54:04.220 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:04.213+0000 7f397948fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:54:04.221 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:04.213+0000 7f397948fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:54:04.221 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:04.213+0000 7f397948fa40 -1 bdev(0x5559a8ecec00 td/osd-mapper/0/block) open stat got: (1) Operation not permitted 2026-02-13T01:54:04.222 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:04.213+0000 7f397948fa40 -1 bluestore(td/osd-mapper/0) _read_fsid unparsable uuid 2026-02-13T01:54:04.955 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T01:54:05.059 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T01:54:05.089 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:05.081+0000 7f7ab5a69a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:54:05.091 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:05.085+0000 7f7ab5a69a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:54:05.092 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:05.085+0000 7f7ab5a69a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:54:05.153 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T01:54:05.627 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:05.621+0000 7f7ab5a69a40 -1 Falling back to public interface 2026-02-13T01:54:06.246 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T01:54:06.635 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:06.629+0000 7f7ab5a69a40 -1 osd.0 0 log_to_monitors true 2026-02-13T01:54:07.337 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T01:54:07.495 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/4272181391,v1:127.0.0.1:6801/4272181391] [v2:127.0.0.1:6802/4272181391,v1:127.0.0.1:6803/4272181391] exists,up fa047f72-1d01-456d-bffe-31a04cd90f71 2026-02-13T01:54:07.500 INFO:tasks.workunit.client.0.trial112.stdout:add osd1 58ef6840-d567-483d-9aea-75b74186097e 2026-02-13T01:54:07.678 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T01:54:07.715 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:07.709+0000 7f31ff3aba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:54:07.716 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:07.709+0000 7f31ff3aba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:54:07.718 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:07.713+0000 7f31ff3aba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:54:07.718 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:07.713+0000 7f31ff3aba40 -1 bdev(0x55712d1dac00 td/osd-mapper/1/block) open stat got: (1) Operation not permitted 2026-02-13T01:54:07.718 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:07.713+0000 7f31ff3aba40 -1 bluestore(td/osd-mapper/1) _read_fsid unparsable uuid 2026-02-13T01:54:08.333 INFO:tasks.workunit.client.0.trial112.stdout:adding osd1 key to auth repository 2026-02-13T01:54:08.548 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T01:54:08.566 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:08.561+0000 7efc983a8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:54:08.568 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:08.561+0000 7efc983a8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:54:08.569 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:08.561+0000 7efc983a8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:54:08.707 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T01:54:08.987 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:08.980+0000 7efc983a8a40 -1 Falling back to public interface 2026-02-13T01:54:09.572 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:09.564+0000 7efc983a8a40 -1 osd.1 0 log_to_monitors true 2026-02-13T01:54:09.866 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T01:54:11.031 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T01:54:11.194 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2005978388,v1:127.0.0.1:6811/2005978388] [v2:127.0.0.1:6812/2005978388,v1:127.0.0.1:6813/2005978388] exists,up 58ef6840-d567-483d-9aea-75b74186097e 2026-02-13T01:54:11.200 INFO:tasks.workunit.client.0.trial112.stdout:add osd2 3394d731-6c3f-4adb-aa2d-101b26c4106b 2026-02-13T01:54:11.378 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T01:54:11.407 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:11.400+0000 7efc096b2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:54:11.408 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:11.400+0000 7efc096b2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:54:11.409 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:11.400+0000 7efc096b2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:54:11.410 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:11.404+0000 7efc096b2a40 -1 bdev(0x563cd0666c00 td/osd-mapper/2/block) open stat got: (1) Operation not permitted 2026-02-13T01:54:11.410 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:11.404+0000 7efc096b2a40 -1 bluestore(td/osd-mapper/2) _read_fsid unparsable uuid 2026-02-13T01:54:12.036 INFO:tasks.workunit.client.0.trial112.stdout:adding osd2 key to auth repository 2026-02-13T01:54:12.244 INFO:tasks.workunit.client.0.trial112.stdout:start osd.2 2026-02-13T01:54:12.273 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:12.264+0000 7f82d43fca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:54:12.274 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:12.268+0000 7f82d43fca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:54:12.275 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:12.268+0000 7f82d43fca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:54:12.400 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T01:54:12.664 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:12.656+0000 7f82d43fca40 -1 Falling back to public interface 2026-02-13T01:54:13.228 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:13.220+0000 7f82d43fca40 -1 osd.2 0 log_to_monitors true 2026-02-13T01:54:13.573 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T01:54:14.726 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T01:54:14.888 INFO:tasks.workunit.client.0.trial112.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3591997503,v1:127.0.0.1:6819/3591997503] [v2:127.0.0.1:6820/3591997503,v1:127.0.0.1:6821/3591997503] exists,up 3394d731-6c3f-4adb-aa2d-101b26c4106b 2026-02-13T01:54:15.101 INFO:tasks.workunit.client.0.trial112.stderr:pool 'test' created 2026-02-13T01:54:16.599 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836490 2026-02-13T01:54:16.775 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705671 2026-02-13T01:54:16.950 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574852 2026-02-13T01:54:18.801 INFO:tasks.workunit.client.0.trial112.stdout:standard_scrub_cluster: dbg: test pool is test 1 2026-02-13T01:54:18.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:55: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_not_scrubbing 1 2026-02-13T01:54:18.865 INFO:tasks.workunit.client.0.trial112.stdout:osd.0: { 2026-02-13T01:54:18.866 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_stats_update_period_not_scrubbing = '' " 2026-02-13T01:54:18.866 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T01:54:18.873 INFO:tasks.workunit.client.0.trial112.stdout:osd.1: { 2026-02-13T01:54:18.873 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_stats_update_period_not_scrubbing = '' " 2026-02-13T01:54:18.873 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T01:54:18.880 INFO:tasks.workunit.client.0.trial112.stdout:osd.2: { 2026-02-13T01:54:18.881 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_stats_update_period_not_scrubbing = '' " 2026-02-13T01:54:18.881 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T01:54:18.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:56: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_scrubbing 1 2026-02-13T01:54:18.967 INFO:tasks.workunit.client.0.trial112.stdout:osd.0: { 2026-02-13T01:54:18.967 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_stats_update_period_scrubbing = '' " 2026-02-13T01:54:18.967 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T01:54:18.974 INFO:tasks.workunit.client.0.trial112.stdout:osd.1: { 2026-02-13T01:54:18.974 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_stats_update_period_scrubbing = '' " 2026-02-13T01:54:18.974 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T01:54:18.981 INFO:tasks.workunit.client.0.trial112.stdout:osd.2: { 2026-02-13T01:54:18.981 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_stats_update_period_scrubbing = '' " 2026-02-13T01:54:18.982 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T01:54:18.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:58: TEST_truncated_sna_record: local osdn=3 2026-02-13T01:54:18.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:59: TEST_truncated_sna_record: local poolid=1 2026-02-13T01:54:18.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:60: TEST_truncated_sna_record: local poolname=test 2026-02-13T01:54:18.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:61: TEST_truncated_sna_record: local objname=objxxx 2026-02-13T01:54:18.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:64: TEST_truncated_sna_record: make_a_clone test objxxx snap01 snap02 2026-02-13T01:54:18.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2026-02-13T01:54:18.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2026-02-13T01:54:19.119 INFO:tasks.workunit.client.0.trial112.stdout:created pool test snap snap01 2026-02-13T01:54:19.220 INFO:tasks.workunit.client.0.trial112.stdout:created pool test snap snap02 2026-02-13T01:54:19.222 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:65: TEST_truncated_sna_record: make_a_clone test objxxx snap13 2026-02-13T01:54:19.222 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2026-02-13T01:54:19.222 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2026-02-13T01:54:19.322 INFO:tasks.workunit.client.0.trial112.stdout:created pool test snap snap13 2026-02-13T01:54:19.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:66: TEST_truncated_sna_record: make_a_clone test objxxx snap24 snap25 2026-02-13T01:54:19.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2026-02-13T01:54:19.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2026-02-13T01:54:19.425 INFO:tasks.workunit.client.0.trial112.stdout:created pool test snap snap24 2026-02-13T01:54:19.526 INFO:tasks.workunit.client.0.trial112.stdout:created pool test snap snap25 2026-02-13T01:54:19.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: echo 13677 2026-02-13T01:54:19.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: rados -p test put objxxx - 2026-02-13T01:54:19.550 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2026-02-13T01:54:19.550 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: jq -r .pgid 2026-02-13T01:54:19.704 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: local pgid=1.3 2026-02-13T01:54:19.705 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2026-02-13T01:54:19.705 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: jq -r '.up[0]' 2026-02-13T01:54:19.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: local osd=1 2026-02-13T01:54:19.862 INFO:tasks.workunit.client.0.trial112.stdout:pgid is 1.3 (primary: osd.1) 2026-02-13T01:54:19.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:72: TEST_truncated_sna_record: echo 'pgid is 1.3 (primary: osd.1)' 2026-02-13T01:54:19.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:74: TEST_truncated_sna_record: set_query_debug 1.3 2026-02-13T01:54:19.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:417: set_query_debug: local pgid=1.3 2026-02-13T01:54:19.863 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:419: set_query_debug: ceph pg dump pgs_brief 2026-02-13T01:54:19.863 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:419: set_query_debug: awk -v 'pg=^1.3' -n -e '$0 ~ pg { print(gensub(/[^0-9]*([0-9]+).*/,"\\1","g",$5)); }' 2026-02-13T01:54:20.014 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs_brief 2026-02-13T01:54:20.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:419: set_query_debug: local prim_osd=1 2026-02-13T01:54:20.025 INFO:tasks.workunit.client.0.trial112.stdout:Setting scrub debug data. Primary for 1.3 is 1 2026-02-13T01:54:20.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:421: set_query_debug: echo 'Setting scrub debug data. Primary for 1.3 is 1' 2026-02-13T01:54:20.027 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:422: set_query_debug: get_asok_path osd.1 2026-02-13T01:54:20.027 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-13T01:54:20.027 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-13T01:54:20.027 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T01:54:20.027 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:54:20.027 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7480 2026-02-13T01:54:20.027 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.7480/ceph-osd.1.asok 2026-02-13T01:54:20.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:422: set_query_debug: CEPH_ARGS= 2026-02-13T01:54:20.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:422: set_query_debug: ceph --format=json daemon /tmp/ceph-asok.7480/ceph-osd.1.asok scrubdebug 1.3 set sessions 2026-02-13T01:54:20.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2026-02-13T01:54:20.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2026-02-13T01:54:20.113 INFO:tasks.workunit.client.0.trial112.stdout:{"success":true}{ 2026-02-13T01:54:20.113 INFO:tasks.workunit.client.0.trial112.stdout: "name": "objxxx", 2026-02-13T01:54:20.114 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 5, 2026-02-13T01:54:20.114 INFO:tasks.workunit.client.0.trial112.stdout: "clones": [ 2026-02-13T01:54:20.114 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T01:54:20.114 INFO:tasks.workunit.client.0.trial112.stdout: "id": 2, 2026-02-13T01:54:20.114 INFO:tasks.workunit.client.0.trial112.stdout: "snapshots": [ 2026-02-13T01:54:20.114 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T01:54:20.114 INFO:tasks.workunit.client.0.trial112.stdout: "id": 1, 2026-02-13T01:54:20.114 INFO:tasks.workunit.client.0.trial112.stdout: "name": "snap01" 2026-02-13T01:54:20.114 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T01:54:20.114 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T01:54:20.114 INFO:tasks.workunit.client.0.trial112.stdout: "id": 2, 2026-02-13T01:54:20.115 INFO:tasks.workunit.client.0.trial112.stdout: "name": "snap02" 2026-02-13T01:54:20.115 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T01:54:20.115 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T01:54:20.115 INFO:tasks.workunit.client.0.trial112.stdout: "size": 5, 2026-02-13T01:54:20.115 INFO:tasks.workunit.client.0.trial112.stdout: "overlaps": [] 2026-02-13T01:54:20.115 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T01:54:20.115 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T01:54:20.115 INFO:tasks.workunit.client.0.trial112.stdout: "id": 3, 2026-02-13T01:54:20.115 INFO:tasks.workunit.client.0.trial112.stdout: "snapshots": [ 2026-02-13T01:54:20.115 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T01:54:20.115 INFO:tasks.workunit.client.0.trial112.stdout: "id": 3, 2026-02-13T01:54:20.115 INFO:tasks.workunit.client.0.trial112.stdout: "name": "snap13" 2026-02-13T01:54:20.116 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T01:54:20.116 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T01:54:20.116 INFO:tasks.workunit.client.0.trial112.stdout: "size": 6, 2026-02-13T01:54:20.116 INFO:tasks.workunit.client.0.trial112.stdout: "overlaps": [] 2026-02-13T01:54:20.116 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T01:54:20.116 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T01:54:20.116 INFO:tasks.workunit.client.0.trial112.stdout: "id": 5, 2026-02-13T01:54:20.116 INFO:tasks.workunit.client.0.trial112.stdout: "snapshots": [ 2026-02-13T01:54:20.116 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T01:54:20.116 INFO:tasks.workunit.client.0.trial112.stdout: "id": 4, 2026-02-13T01:54:20.116 INFO:tasks.workunit.client.0.trial112.stdout: "name": "snap24" 2026-02-13T01:54:20.116 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T01:54:20.117 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T01:54:20.117 INFO:tasks.workunit.client.0.trial112.stdout: "id": 5, 2026-02-13T01:54:20.117 INFO:tasks.workunit.client.0.trial112.stdout: "name": "snap25" 2026-02-13T01:54:20.117 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T01:54:20.117 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T01:54:20.117 INFO:tasks.workunit.client.0.trial112.stdout: "size": 6, 2026-02-13T01:54:20.117 INFO:tasks.workunit.client.0.trial112.stdout: "overlaps": [] 2026-02-13T01:54:20.117 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T01:54:20.117 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T01:54:20.117 INFO:tasks.workunit.client.0.trial112.stdout: "id": "head", 2026-02-13T01:54:20.117 INFO:tasks.workunit.client.0.trial112.stdout: "snapshots": [], 2026-02-13T01:54:20.117 INFO:tasks.workunit.client.0.trial112.stdout: "size": 6 2026-02-13T01:54:20.117 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T01:54:20.118 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T01:54:20.118 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T01:54:20.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:80: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2026-02-13T01:54:20.182 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T01:54:20.182 INFO:tasks.workunit.client.0.trial112.stdout: "deep": true, 2026-02-13T01:54:20.182 INFO:tasks.workunit.client.0.trial112.stdout: "must": true, 2026-02-13T01:54:20.182 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "0.000000" 2026-02-13T01:54:20.182 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T01:54:20.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:84: TEST_truncated_sna_record: sleep 3 2026-02-13T01:54:23.190 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:85: TEST_truncated_sna_record: ceph pg dump pgs 2026-02-13T01:54:23.337 INFO:tasks.workunit.client.0.trial112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-13T01:54:23.337 INFO:tasks.workunit.client.0.trial112.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2026-02-13T01:54:20.588390+0000 22'7 22:43 [1,2,0] 1 [1,2,0] 1 22'7 2026-02-13T01:54:20.588325+0000 22'7 2026-02-13T01:54:20.588325+0000 0 0 periodic scrub scheduled @ 2026-02-14T01:54:20.588325+0000 4 0 2026-02-13T01:54:23.337 INFO:tasks.workunit.client.0.trial112.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-13T01:54:15.205743+0000 0'0 22:30 [0,1,2] 0 [0,1,2] 0 0'0 2026-02-13T01:54:15.096694+0000 0'0 2026-02-13T01:54:15.096694+0000 0 0 periodic scrub scheduled @ 2026-02-14T01:54:15.096694+0000 0 0 2026-02-13T01:54:23.337 INFO:tasks.workunit.client.0.trial112.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-13T01:54:15.613167+0000 0'0 22:28 [2,0,1] 2 [2,0,1] 2 0'0 2026-02-13T01:54:15.096694+0000 0'0 2026-02-13T01:54:15.096694+0000 0 0 periodic scrub scheduled @ 2026-02-14T01:54:15.096694+0000 0 0 2026-02-13T01:54:23.337 INFO:tasks.workunit.client.0.trial112.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-13T01:54:15.206354+0000 0'0 22:31 [1,0,2] 1 [1,0,2] 1 0'0 2026-02-13T01:54:15.096694+0000 0'0 2026-02-13T01:54:15.096694+0000 0 0 periodic scrub scheduled @ 2026-02-14T01:54:15.096694+0000 0 0 2026-02-13T01:54:23.338 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:23.338 INFO:tasks.workunit.client.0.trial112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-13T01:54:23.338 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T01:54:23.347 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:86: TEST_truncated_sna_record: grep -a -q -- 'event: --^^^^---- ScrubFinished' td/osd-mapper/osd.1.log 2026-02-13T01:54:23.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:90: TEST_truncated_sna_record: ceph pg dump pgs 2026-02-13T01:54:23.508 INFO:tasks.workunit.client.0.trial112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-13T01:54:23.509 INFO:tasks.workunit.client.0.trial112.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2026-02-13T01:54:20.588390+0000 22'7 22:43 [1,2,0] 1 [1,2,0] 1 22'7 2026-02-13T01:54:20.588325+0000 22'7 2026-02-13T01:54:20.588325+0000 0 0 periodic scrub scheduled @ 2026-02-14T01:54:20.588325+0000 4 0 2026-02-13T01:54:23.509 INFO:tasks.workunit.client.0.trial112.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-13T01:54:15.205743+0000 0'0 22:30 [0,1,2] 0 [0,1,2] 0 0'0 2026-02-13T01:54:15.096694+0000 0'0 2026-02-13T01:54:15.096694+0000 0 0 periodic scrub scheduled @ 2026-02-14T01:54:15.096694+0000 0 0 2026-02-13T01:54:23.509 INFO:tasks.workunit.client.0.trial112.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-13T01:54:15.613167+0000 0'0 22:28 [2,0,1] 2 [2,0,1] 2 0'0 2026-02-13T01:54:15.096694+0000 0'0 2026-02-13T01:54:15.096694+0000 0 0 periodic scrub scheduled @ 2026-02-14T01:54:15.096694+0000 0 0 2026-02-13T01:54:23.509 INFO:tasks.workunit.client.0.trial112.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-13T01:54:15.206354+0000 0'0 22:31 [1,0,2] 1 [1,0,2] 1 0'0 2026-02-13T01:54:15.096694+0000 0'0 2026-02-13T01:54:15.096694+0000 0 0 periodic scrub scheduled @ 2026-02-14T01:54:15.096694+0000 0 0 2026-02-13T01:54:23.509 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:23.509 INFO:tasks.workunit.client.0.trial112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-13T01:54:23.510 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T01:54:23.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:91: TEST_truncated_sna_record: ceph osd set noscrub 2026-02-13T01:54:23.735 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T01:54:23.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:92: TEST_truncated_sna_record: ceph osd set nodeep-scrub 2026-02-13T01:54:23.940 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T01:54:23.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:93: TEST_truncated_sna_record: sleep 5 2026-02-13T01:54:28.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:94: TEST_truncated_sna_record: grep -a -q -v ERR td/osd-mapper/osd.1.log 2026-02-13T01:54:28.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:97: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2026-02-13T01:54:28.955 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T01:54:28.955 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T01:54:28.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T01:54:28.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T01:54:28.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T01:54:29.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T01:54:29.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2026-02-13T01:54:29.266 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2026-02-13T01:54:29.279 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.272+0000 7f6067642d40 1 bdev(0x55fa47695800 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2026-02-13T01:54:29.280 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.272+0000 7f6067642d40 1 bdev(0x55fa47695800 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-13T01:54:29.280 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.272+0000 7f6067642d40 1 bdev(0x55fa47695800 td/osd-mapper/0/block) close 2026-02-13T01:54:29.330 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.324+0000 7f6067642d40 1 bdev(0x55fa47695800 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2026-02-13T01:54:29.330 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.324+0000 7f6067642d40 1 bdev(0x55fa47695800 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-13T01:54:29.331 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.324+0000 7f6067642d40 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-13T01:54:29.331 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.324+0000 7f6067642d40 1 bdev(0x55fa47695c00 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2026-02-13T01:54:29.332 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.324+0000 7f6067642d40 1 bdev(0x55fa47695c00 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-13T01:54:29.332 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.324+0000 7f6067642d40 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2026-02-13T01:54:29.332 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.324+0000 7f6067642d40 1 bluefs mount 2026-02-13T01:54:29.332 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.324+0000 7f6067642d40 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-13T01:54:29.334 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.324+0000 7f6067642d40 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:29.334 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.324+0000 7f6067642d40 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:29.334 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.324+0000 7f6067642d40 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:29.334 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.324+0000 7f6067642d40 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:29.334 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.324+0000 7f6067642d40 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:29.334 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.324+0000 7f6067642d40 1 bluefs mount shared_bdev_used = 0 2026-02-13T01:54:29.335 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.324+0000 7f6067642d40 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-13T01:54:29.346 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.336+0000 7f6067642d40 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-13T01:54:29.346 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.336+0000 7f6067642d40 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2051 2026-02-13T01:54:29.346 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.336+0000 7f6067642d40 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2026-02-13T01:54:29.346 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.336+0000 7f6067642d40 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-13T01:54:29.347 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.336+0000 7f6067642d40 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2026-02-13T01:54:29.347 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.340+0000 7f6067642d40 1 freelist init 2026-02-13T01:54:29.347 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.340+0000 7f6067642d40 1 freelist _read_cfg 2026-02-13T01:54:29.347 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.340+0000 7f6067642d40 1 bluestore(td/osd-mapper/0) _open_fm effective freelist_type = null, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2026-02-13T01:54:29.348 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.340+0000 7f6067642d40 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 19 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc8000, fragmentation 6.86647e-07 2026-02-13T01:54:29.348 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.340+0000 7f6067642d40 1 bluefs umount 2026-02-13T01:54:29.348 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.340+0000 7f6067642d40 1 bdev(0x55fa47695c00 td/osd-mapper/0/block) close 2026-02-13T01:54:29.398 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.392+0000 7f6067642d40 1 bdev(0x55fa47695c00 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2026-02-13T01:54:29.399 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.392+0000 7f6067642d40 1 bdev(0x55fa47695c00 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-13T01:54:29.399 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.392+0000 7f6067642d40 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2026-02-13T01:54:29.399 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.392+0000 7f6067642d40 1 bluefs mount 2026-02-13T01:54:29.399 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.392+0000 7f6067642d40 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-13T01:54:29.400 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.392+0000 7f6067642d40 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:29.400 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.392+0000 7f6067642d40 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:29.400 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.392+0000 7f6067642d40 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:29.400 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.392+0000 7f6067642d40 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:29.400 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.392+0000 7f6067642d40 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:29.400 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.392+0000 7f6067642d40 1 bluefs mount shared_bdev_used = 27656192 2026-02-13T01:54:29.401 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.392+0000 7f6067642d40 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-13T01:54:29.411 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.404+0000 7f6067642d40 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-13T01:54:29.412 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2026-02-13T01:54:29.412 INFO:tasks.workunit.client.0.trial112.stdout:00000000 0b 05 58 02 00 00 b7 9c f4 58 12 42 4c c6 8f 9e |..X......X.BL...| 2026-02-13T01:54:29.412 INFO:tasks.workunit.client.0.trial112.stdout:00000010 48 1a 9a ef bd ea 00 00 00 00 18 00 00 00 00 00 |H...............| 2026-02-13T01:54:29.413 INFO:tasks.workunit.client.0.trial112.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2026-02-13T01:54:29.413 INFO:tasks.workunit.client.0.trial112.stdout:* 2026-02-13T01:54:29.413 INFO:tasks.workunit.client.0.trial112.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2026-02-13T01:54:29.413 INFO:tasks.workunit.client.0.trial112.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2026-02-13T01:54:29.413 INFO:tasks.workunit.client.0.trial112.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2026-02-13T01:54:29.413 INFO:tasks.workunit.client.0.trial112.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2026-02-13T01:54:29.413 INFO:tasks.workunit.client.0.trial112.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2026-02-13T01:54:29.413 INFO:tasks.workunit.client.0.trial112.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2026-02-13T01:54:29.413 INFO:tasks.workunit.client.0.trial112.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2026-02-13T01:54:29.413 INFO:tasks.workunit.client.0.trial112.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2026-02-13T01:54:29.413 INFO:tasks.workunit.client.0.trial112.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2026-02-13T01:54:29.414 INFO:tasks.workunit.client.0.trial112.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2026-02-13T01:54:29.414 INFO:tasks.workunit.client.0.trial112.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2026-02-13T01:54:29.414 INFO:tasks.workunit.client.0.trial112.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2026-02-13T01:54:29.414 INFO:tasks.workunit.client.0.trial112.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2026-02-13T01:54:29.414 INFO:tasks.workunit.client.0.trial112.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2026-02-13T01:54:29.414 INFO:tasks.workunit.client.0.trial112.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2026-02-13T01:54:29.414 INFO:tasks.workunit.client.0.trial112.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2026-02-13T01:54:29.414 INFO:tasks.workunit.client.0.trial112.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2026-02-13T01:54:29.414 INFO:tasks.workunit.client.0.trial112.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2026-02-13T01:54:29.414 INFO:tasks.workunit.client.0.trial112.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2026-02-13T01:54:29.414 INFO:tasks.workunit.client.0.trial112.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2026-02-13T01:54:29.414 INFO:tasks.workunit.client.0.trial112.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2026-02-13T01:54:29.415 INFO:tasks.workunit.client.0.trial112.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2026-02-13T01:54:29.415 INFO:tasks.workunit.client.0.trial112.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2026-02-13T01:54:29.415 INFO:tasks.workunit.client.0.trial112.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2026-02-13T01:54:29.415 INFO:tasks.workunit.client.0.trial112.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2026-02-13T01:54:29.415 INFO:tasks.workunit.client.0.trial112.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2026-02-13T01:54:29.415 INFO:tasks.workunit.client.0.trial112.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2026-02-13T01:54:29.415 INFO:tasks.workunit.client.0.trial112.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2026-02-13T01:54:29.415 INFO:tasks.workunit.client.0.trial112.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2026-02-13T01:54:29.415 INFO:tasks.workunit.client.0.trial112.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2026-02-13T01:54:29.415 INFO:tasks.workunit.client.0.trial112.stdout:00000220 72 65 18 00 00 00 05 00 00 00 fa 04 7f 72 1d 01 |re...........r..| 2026-02-13T01:54:29.415 INFO:tasks.workunit.client.0.trial112.stdout:00000230 45 6d bf fe 31 a0 4c d9 0f 71 00 00 00 00 00 00 |Em..1.L..q......| 2026-02-13T01:54:29.415 INFO:tasks.workunit.client.0.trial112.stdout:00000240 00 00 18 00 00 00 3f 84 8e 69 46 55 0f 23 01 00 |......?..iFU.#..| 2026-02-13T01:54:29.416 INFO:tasks.workunit.client.0.trial112.stdout:00000250 00 00 01 00 00 00 01 00 00 00 18 00 00 00 |..............| 2026-02-13T01:54:29.416 INFO:tasks.workunit.client.0.trial112.stdout:0000025e 2026-02-13T01:54:29.416 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:29.416 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2026-02-13T01:54:29.416 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:29.416 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2026-02-13T01:54:29.416 INFO:tasks.workunit.client.0.trial112.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2026-02-13T01:54:29.416 INFO:tasks.workunit.client.0.trial112.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2026-02-13T01:54:29.416 INFO:tasks.workunit.client.0.trial112.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-13T01:54:29.416 INFO:tasks.workunit.client.0.trial112.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2026-02-13T01:54:29.416 INFO:tasks.workunit.client.0.trial112.stdout:00000040 00 00 00 00 00 00 00 |.......| 2026-02-13T01:54:29.416 INFO:tasks.workunit.client.0.trial112.stdout:00000047 2026-02-13T01:54:29.417 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:29.417 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2026-02-13T01:54:29.417 INFO:tasks.workunit.client.0.trial112.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2026-02-13T01:54:29.417 INFO:tasks.workunit.client.0.trial112.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2026-02-13T01:54:29.417 INFO:tasks.workunit.client.0.trial112.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-13T01:54:29.417 INFO:tasks.workunit.client.0.trial112.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2026-02-13T01:54:29.417 INFO:tasks.workunit.client.0.trial112.stdout:0000003f 2026-02-13T01:54:29.417 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:29.417 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2026-02-13T01:54:29.417 INFO:tasks.workunit.client.0.trial112.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2026-02-13T01:54:29.417 INFO:tasks.workunit.client.0.trial112.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2026-02-13T01:54:29.418 INFO:tasks.workunit.client.0.trial112.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-13T01:54:29.418 INFO:tasks.workunit.client.0.trial112.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2026-02-13T01:54:29.418 INFO:tasks.workunit.client.0.trial112.stdout:00000040 00 00 00 00 00 00 00 |.......| 2026-02-13T01:54:29.418 INFO:tasks.workunit.client.0.trial112.stdout:00000047 2026-02-13T01:54:29.418 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:29.418 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2026-02-13T01:54:29.418 INFO:tasks.workunit.client.0.trial112.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-13T01:54:29.418 INFO:tasks.workunit.client.0.trial112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-13T01:54:29.418 INFO:tasks.workunit.client.0.trial112.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-13T01:54:29.418 INFO:tasks.workunit.client.0.trial112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-13T01:54:29.418 INFO:tasks.workunit.client.0.trial112.stdout:0000003b 2026-02-13T01:54:29.418 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:29.418 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2026-02-13T01:54:29.419 INFO:tasks.workunit.client.0.trial112.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-13T01:54:29.419 INFO:tasks.workunit.client.0.trial112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-13T01:54:29.419 INFO:tasks.workunit.client.0.trial112.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-13T01:54:29.419 INFO:tasks.workunit.client.0.trial112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-13T01:54:29.419 INFO:tasks.workunit.client.0.trial112.stdout:0000003b 2026-02-13T01:54:29.419 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:29.419 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-13T01:54:29.419 INFO:tasks.workunit.client.0.trial112.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-13T01:54:29.419 INFO:tasks.workunit.client.0.trial112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-13T01:54:29.419 INFO:tasks.workunit.client.0.trial112.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-13T01:54:29.419 INFO:tasks.workunit.client.0.trial112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-13T01:54:29.419 INFO:tasks.workunit.client.0.trial112.stdout:0000003b 2026-02-13T01:54:29.420 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:29.420 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2026-02-13T01:54:29.420 INFO:tasks.workunit.client.0.trial112.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-13T01:54:29.420 INFO:tasks.workunit.client.0.trial112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-13T01:54:29.420 INFO:tasks.workunit.client.0.trial112.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-13T01:54:29.420 INFO:tasks.workunit.client.0.trial112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-13T01:54:29.420 INFO:tasks.workunit.client.0.trial112.stdout:0000003b 2026-02-13T01:54:29.420 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:29.420 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2026-02-13T01:54:29.420 INFO:tasks.workunit.client.0.trial112.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-13T01:54:29.420 INFO:tasks.workunit.client.0.trial112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-13T01:54:29.420 INFO:tasks.workunit.client.0.trial112.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-13T01:54:29.420 INFO:tasks.workunit.client.0.trial112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-13T01:54:29.421 INFO:tasks.workunit.client.0.trial112.stdout:0000003b 2026-02-13T01:54:29.421 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:29.421 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2026-02-13T01:54:29.421 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:29.421 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2026-02-13T01:54:29.421 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:29.421 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.404+0000 7f6067642d40 1 bluefs umount 2026-02-13T01:54:29.422 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.404+0000 7f6067642d40 1 bdev(0x55fa47695c00 td/osd-mapper/0/block) close 2026-02-13T01:54:29.462 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.456+0000 7f6067642d40 1 freelist shutdown 2026-02-13T01:54:29.462 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.456+0000 7f6067642d40 1 bdev(0x55fa47695800 td/osd-mapper/0/block) close 2026-02-13T01:54:29.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2026-02-13T01:54:29.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2026-02-13T01:54:29.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: grep -a SNA_ 2026-02-13T01:54:29.530 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.524+0000 7f741631dd40 1 bdev(0x564c10091800 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2026-02-13T01:54:29.530 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.524+0000 7f741631dd40 1 bdev(0x564c10091800 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-13T01:54:29.531 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.524+0000 7f741631dd40 1 bdev(0x564c10091800 td/osd-mapper/2/block) close 2026-02-13T01:54:29.570 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.564+0000 7f741631dd40 1 bdev(0x564c10091800 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2026-02-13T01:54:29.570 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.564+0000 7f741631dd40 1 bdev(0x564c10091800 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-13T01:54:29.571 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.564+0000 7f741631dd40 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-13T01:54:29.571 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.564+0000 7f741631dd40 1 bdev(0x564c10091c00 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2026-02-13T01:54:29.571 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.564+0000 7f741631dd40 1 bdev(0x564c10091c00 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-13T01:54:29.572 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.564+0000 7f741631dd40 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2026-02-13T01:54:29.572 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.564+0000 7f741631dd40 1 bluefs mount 2026-02-13T01:54:29.572 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.564+0000 7f741631dd40 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-13T01:54:29.574 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.568+0000 7f741631dd40 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:29.574 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.568+0000 7f741631dd40 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:29.574 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.568+0000 7f741631dd40 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:29.575 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.568+0000 7f741631dd40 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:29.575 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.568+0000 7f741631dd40 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:29.575 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.568+0000 7f741631dd40 1 bluefs mount shared_bdev_used = 0 2026-02-13T01:54:29.575 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.568+0000 7f741631dd40 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-13T01:54:29.587 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.580+0000 7f741631dd40 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-13T01:54:29.587 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.580+0000 7f741631dd40 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2051 2026-02-13T01:54:29.587 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.580+0000 7f741631dd40 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2026-02-13T01:54:29.588 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.580+0000 7f741631dd40 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-13T01:54:29.588 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.580+0000 7f741631dd40 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2026-02-13T01:54:29.588 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.580+0000 7f741631dd40 1 freelist init 2026-02-13T01:54:29.588 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.580+0000 7f741631dd40 1 freelist _read_cfg 2026-02-13T01:54:29.588 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.580+0000 7f741631dd40 1 bluestore(td/osd-mapper/2) _open_fm effective freelist_type = null, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2026-02-13T01:54:29.589 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.580+0000 7f741631dd40 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 12 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc6000, fragmentation 4.19618e-07 2026-02-13T01:54:29.589 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.580+0000 7f741631dd40 1 bluefs umount 2026-02-13T01:54:29.589 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.580+0000 7f741631dd40 1 bdev(0x564c10091c00 td/osd-mapper/2/block) close 2026-02-13T01:54:29.646 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.640+0000 7f741631dd40 1 bdev(0x564c10091c00 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2026-02-13T01:54:29.647 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.640+0000 7f741631dd40 1 bdev(0x564c10091c00 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-13T01:54:29.647 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.640+0000 7f741631dd40 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2026-02-13T01:54:29.647 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.640+0000 7f741631dd40 1 bluefs mount 2026-02-13T01:54:29.647 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.640+0000 7f741631dd40 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-13T01:54:29.648 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.640+0000 7f741631dd40 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:29.648 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.640+0000 7f741631dd40 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:29.649 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.640+0000 7f741631dd40 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:29.649 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.640+0000 7f741631dd40 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:29.649 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.640+0000 7f741631dd40 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:29.649 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.640+0000 7f741631dd40 1 bluefs mount shared_bdev_used = 27656192 2026-02-13T01:54:29.649 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.640+0000 7f741631dd40 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-13T01:54:29.660 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.652+0000 7f741631dd40 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-13T01:54:29.660 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.652+0000 7f741631dd40 1 bluefs umount 2026-02-13T01:54:29.660 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.652+0000 7f741631dd40 1 bdev(0x564c10091c00 td/osd-mapper/2/block) close 2026-02-13T01:54:29.706 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.700+0000 7f741631dd40 1 freelist shutdown 2026-02-13T01:54:29.706 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:29.700+0000 7f741631dd40 1 bdev(0x564c10091800 td/osd-mapper/2/block) close 2026-02-13T01:54:29.754 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2026-02-13T01:54:29.754 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2026-02-13T01:54:29.754 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-13T01:54:29.754 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2026-02-13T01:54:29.754 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2026-02-13T01:54:29.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2026-02-13T01:54:29.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: grep -a SNA_ /tmp/oo2.dump 2026-02-13T01:54:29.755 INFO:tasks.workunit.client.0.trial112.stderr:grep: /tmp/oo2.dump: No such file or directory 2026-02-13T01:54:29.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2026-02-13T01:54:29.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2026-02-13T01:54:29.905 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2026-02-13T01:54:29.905 INFO:tasks.workunit.client.0.trial112.stdout:00000000 0b 05 58 02 00 00 b7 9c f4 58 12 42 4c c6 8f 9e |..X......X.BL...| 2026-02-13T01:54:29.906 INFO:tasks.workunit.client.0.trial112.stdout:00000010 48 1a 9a ef bd ea 02 00 00 00 19 00 00 00 00 00 |H...............| 2026-02-13T01:54:29.906 INFO:tasks.workunit.client.0.trial112.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2026-02-13T01:54:29.906 INFO:tasks.workunit.client.0.trial112.stdout:* 2026-02-13T01:54:29.906 INFO:tasks.workunit.client.0.trial112.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2026-02-13T01:54:29.906 INFO:tasks.workunit.client.0.trial112.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2026-02-13T01:54:29.906 INFO:tasks.workunit.client.0.trial112.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2026-02-13T01:54:29.906 INFO:tasks.workunit.client.0.trial112.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2026-02-13T01:54:29.907 INFO:tasks.workunit.client.0.trial112.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2026-02-13T01:54:29.907 INFO:tasks.workunit.client.0.trial112.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2026-02-13T01:54:29.907 INFO:tasks.workunit.client.0.trial112.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2026-02-13T01:54:29.907 INFO:tasks.workunit.client.0.trial112.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2026-02-13T01:54:29.907 INFO:tasks.workunit.client.0.trial112.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2026-02-13T01:54:29.907 INFO:tasks.workunit.client.0.trial112.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2026-02-13T01:54:29.907 INFO:tasks.workunit.client.0.trial112.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2026-02-13T01:54:29.908 INFO:tasks.workunit.client.0.trial112.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2026-02-13T01:54:29.908 INFO:tasks.workunit.client.0.trial112.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2026-02-13T01:54:29.908 INFO:tasks.workunit.client.0.trial112.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2026-02-13T01:54:29.908 INFO:tasks.workunit.client.0.trial112.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2026-02-13T01:54:29.908 INFO:tasks.workunit.client.0.trial112.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2026-02-13T01:54:29.908 INFO:tasks.workunit.client.0.trial112.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2026-02-13T01:54:29.908 INFO:tasks.workunit.client.0.trial112.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2026-02-13T01:54:29.909 INFO:tasks.workunit.client.0.trial112.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2026-02-13T01:54:29.909 INFO:tasks.workunit.client.0.trial112.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2026-02-13T01:54:29.909 INFO:tasks.workunit.client.0.trial112.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2026-02-13T01:54:29.909 INFO:tasks.workunit.client.0.trial112.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2026-02-13T01:54:29.909 INFO:tasks.workunit.client.0.trial112.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2026-02-13T01:54:29.909 INFO:tasks.workunit.client.0.trial112.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2026-02-13T01:54:29.909 INFO:tasks.workunit.client.0.trial112.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2026-02-13T01:54:29.909 INFO:tasks.workunit.client.0.trial112.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2026-02-13T01:54:29.910 INFO:tasks.workunit.client.0.trial112.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2026-02-13T01:54:29.910 INFO:tasks.workunit.client.0.trial112.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2026-02-13T01:54:29.910 INFO:tasks.workunit.client.0.trial112.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2026-02-13T01:54:29.910 INFO:tasks.workunit.client.0.trial112.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2026-02-13T01:54:29.910 INFO:tasks.workunit.client.0.trial112.stdout:00000220 72 65 19 00 00 00 0d 00 00 00 33 94 d7 31 6c 3f |re........3..1l?| 2026-02-13T01:54:29.910 INFO:tasks.workunit.client.0.trial112.stdout:00000230 4a db aa 2d 10 1b 26 c4 10 6b 00 00 00 00 00 00 |J..-..&..k......| 2026-02-13T01:54:29.910 INFO:tasks.workunit.client.0.trial112.stdout:00000240 00 00 19 00 00 00 46 84 8e 69 81 17 58 10 01 00 |......F..i..X...| 2026-02-13T01:54:29.911 INFO:tasks.workunit.client.0.trial112.stdout:00000250 00 00 01 00 00 00 01 00 00 00 19 00 00 00 |..............| 2026-02-13T01:54:29.911 INFO:tasks.workunit.client.0.trial112.stdout:0000025e 2026-02-13T01:54:29.911 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:29.911 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2026-02-13T01:54:29.911 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:29.911 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2026-02-13T01:54:29.911 INFO:tasks.workunit.client.0.trial112.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2026-02-13T01:54:29.912 INFO:tasks.workunit.client.0.trial112.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2026-02-13T01:54:29.912 INFO:tasks.workunit.client.0.trial112.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-13T01:54:29.912 INFO:tasks.workunit.client.0.trial112.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2026-02-13T01:54:29.912 INFO:tasks.workunit.client.0.trial112.stdout:00000040 00 00 00 00 00 00 00 |.......| 2026-02-13T01:54:29.912 INFO:tasks.workunit.client.0.trial112.stdout:00000047 2026-02-13T01:54:29.912 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:29.912 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2026-02-13T01:54:29.912 INFO:tasks.workunit.client.0.trial112.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2026-02-13T01:54:29.913 INFO:tasks.workunit.client.0.trial112.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2026-02-13T01:54:29.913 INFO:tasks.workunit.client.0.trial112.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-13T01:54:29.913 INFO:tasks.workunit.client.0.trial112.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2026-02-13T01:54:29.913 INFO:tasks.workunit.client.0.trial112.stdout:0000003f 2026-02-13T01:54:29.913 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:29.913 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2026-02-13T01:54:29.913 INFO:tasks.workunit.client.0.trial112.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2026-02-13T01:54:29.914 INFO:tasks.workunit.client.0.trial112.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2026-02-13T01:54:29.914 INFO:tasks.workunit.client.0.trial112.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-13T01:54:29.914 INFO:tasks.workunit.client.0.trial112.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2026-02-13T01:54:29.914 INFO:tasks.workunit.client.0.trial112.stdout:00000040 00 00 00 00 00 00 00 |.......| 2026-02-13T01:54:29.914 INFO:tasks.workunit.client.0.trial112.stdout:00000047 2026-02-13T01:54:29.914 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:29.914 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2026-02-13T01:54:29.914 INFO:tasks.workunit.client.0.trial112.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-13T01:54:29.915 INFO:tasks.workunit.client.0.trial112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-13T01:54:29.915 INFO:tasks.workunit.client.0.trial112.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-13T01:54:29.915 INFO:tasks.workunit.client.0.trial112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-13T01:54:29.915 INFO:tasks.workunit.client.0.trial112.stdout:0000003b 2026-02-13T01:54:29.915 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:29.915 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2026-02-13T01:54:29.915 INFO:tasks.workunit.client.0.trial112.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-13T01:54:29.916 INFO:tasks.workunit.client.0.trial112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-13T01:54:29.916 INFO:tasks.workunit.client.0.trial112.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-13T01:54:29.916 INFO:tasks.workunit.client.0.trial112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-13T01:54:29.916 INFO:tasks.workunit.client.0.trial112.stdout:0000003b 2026-02-13T01:54:29.916 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:29.916 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-13T01:54:29.916 INFO:tasks.workunit.client.0.trial112.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-13T01:54:29.917 INFO:tasks.workunit.client.0.trial112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-13T01:54:29.917 INFO:tasks.workunit.client.0.trial112.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-13T01:54:29.917 INFO:tasks.workunit.client.0.trial112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-13T01:54:29.917 INFO:tasks.workunit.client.0.trial112.stdout:0000003b 2026-02-13T01:54:29.917 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:29.917 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2026-02-13T01:54:29.917 INFO:tasks.workunit.client.0.trial112.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-13T01:54:29.917 INFO:tasks.workunit.client.0.trial112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-13T01:54:29.918 INFO:tasks.workunit.client.0.trial112.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-13T01:54:29.918 INFO:tasks.workunit.client.0.trial112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-13T01:54:29.918 INFO:tasks.workunit.client.0.trial112.stdout:0000003b 2026-02-13T01:54:29.918 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:29.918 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2026-02-13T01:54:29.918 INFO:tasks.workunit.client.0.trial112.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-13T01:54:29.918 INFO:tasks.workunit.client.0.trial112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-13T01:54:29.918 INFO:tasks.workunit.client.0.trial112.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-13T01:54:29.919 INFO:tasks.workunit.client.0.trial112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-13T01:54:29.919 INFO:tasks.workunit.client.0.trial112.stdout:0000003b 2026-02-13T01:54:29.919 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:29.919 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2026-02-13T01:54:29.919 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:29.919 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2026-02-13T01:54:29.919 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:29.986 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2026-02-13T01:54:29.986 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2026-02-13T01:54:29.986 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: awk -e '{print $2;}' 2026-02-13T01:54:29.987 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: wc -l 2026-02-13T01:54:30.228 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: local num_sna_b4=5 2026-02-13T01:54:30.229 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: expr 3 - 1 2026-02-13T01:54:30.230 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: seq 0 2 2026-02-13T01:54:30.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2026-02-13T01:54:30.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/0 2026-02-13T01:54:30.232 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0)' 2026-02-13T01:54:30.232 INFO:tasks.workunit.client.0.trial112.stdout:corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0) 2026-02-13T01:54:30.232 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2026-02-13T01:54:30.232 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2026-02-13T01:54:30.246 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.236+0000 7faabfa6cd40 1 bdev(0x5600fab81800 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2026-02-13T01:54:30.246 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.236+0000 7faabfa6cd40 1 bdev(0x5600fab81800 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-13T01:54:30.246 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.236+0000 7faabfa6cd40 1 bdev(0x5600fab81800 td/osd-mapper/0/block) close 2026-02-13T01:54:30.290 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.284+0000 7faabfa6cd40 1 bdev(0x5600fab81800 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2026-02-13T01:54:30.290 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.284+0000 7faabfa6cd40 1 bdev(0x5600fab81800 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-13T01:54:30.292 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.284+0000 7faabfa6cd40 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-13T01:54:30.292 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.284+0000 7faabfa6cd40 1 bdev(0x5600fab81c00 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2026-02-13T01:54:30.292 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.284+0000 7faabfa6cd40 1 bdev(0x5600fab81c00 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-13T01:54:30.292 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.284+0000 7faabfa6cd40 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2026-02-13T01:54:30.293 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.284+0000 7faabfa6cd40 1 bluefs mount 2026-02-13T01:54:30.293 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.284+0000 7faabfa6cd40 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-13T01:54:30.295 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.288+0000 7faabfa6cd40 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:30.295 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.288+0000 7faabfa6cd40 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:30.295 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.288+0000 7faabfa6cd40 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:30.295 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.288+0000 7faabfa6cd40 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:30.295 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.288+0000 7faabfa6cd40 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:30.295 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.288+0000 7faabfa6cd40 1 bluefs mount shared_bdev_used = 0 2026-02-13T01:54:30.295 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.288+0000 7faabfa6cd40 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-13T01:54:30.307 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.300+0000 7faabfa6cd40 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-13T01:54:30.307 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.300+0000 7faabfa6cd40 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2051 2026-02-13T01:54:30.307 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.300+0000 7faabfa6cd40 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2026-02-13T01:54:30.307 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.300+0000 7faabfa6cd40 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-13T01:54:30.308 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.300+0000 7faabfa6cd40 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2026-02-13T01:54:30.308 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.300+0000 7faabfa6cd40 1 freelist init 2026-02-13T01:54:30.308 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.300+0000 7faabfa6cd40 1 freelist _read_cfg 2026-02-13T01:54:30.308 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.300+0000 7faabfa6cd40 1 bluestore(td/osd-mapper/0) _open_fm effective freelist_type = null, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2026-02-13T01:54:30.309 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.300+0000 7faabfa6cd40 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 19 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc8000, fragmentation 6.86647e-07 2026-02-13T01:54:30.309 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.300+0000 7faabfa6cd40 1 bluefs umount 2026-02-13T01:54:30.310 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.300+0000 7faabfa6cd40 1 bdev(0x5600fab81c00 td/osd-mapper/0/block) close 2026-02-13T01:54:30.351 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.344+0000 7faabfa6cd40 1 bdev(0x5600fab81c00 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2026-02-13T01:54:30.351 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.344+0000 7faabfa6cd40 1 bdev(0x5600fab81c00 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-13T01:54:30.351 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.344+0000 7faabfa6cd40 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2026-02-13T01:54:30.352 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.344+0000 7faabfa6cd40 1 bluefs mount 2026-02-13T01:54:30.352 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.344+0000 7faabfa6cd40 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-13T01:54:30.352 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.344+0000 7faabfa6cd40 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:30.352 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.344+0000 7faabfa6cd40 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:30.353 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.344+0000 7faabfa6cd40 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:30.353 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.344+0000 7faabfa6cd40 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:30.353 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.344+0000 7faabfa6cd40 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:30.353 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.344+0000 7faabfa6cd40 1 bluefs mount shared_bdev_used = 27656192 2026-02-13T01:54:30.353 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.344+0000 7faabfa6cd40 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-13T01:54:30.363 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.356+0000 7faabfa6cd40 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-13T01:54:30.365 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2026-02-13T01:54:30.365 INFO:tasks.workunit.client.0.trial112.stdout:00000000 0b 05 58 02 00 00 b7 9c f4 58 12 42 4c c6 8f 9e |..X......X.BL...| 2026-02-13T01:54:30.365 INFO:tasks.workunit.client.0.trial112.stdout:00000010 48 1a 9a ef bd ea 00 00 00 00 18 00 00 00 00 00 |H...............| 2026-02-13T01:54:30.365 INFO:tasks.workunit.client.0.trial112.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2026-02-13T01:54:30.365 INFO:tasks.workunit.client.0.trial112.stdout:* 2026-02-13T01:54:30.365 INFO:tasks.workunit.client.0.trial112.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2026-02-13T01:54:30.365 INFO:tasks.workunit.client.0.trial112.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2026-02-13T01:54:30.365 INFO:tasks.workunit.client.0.trial112.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2026-02-13T01:54:30.365 INFO:tasks.workunit.client.0.trial112.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2026-02-13T01:54:30.365 INFO:tasks.workunit.client.0.trial112.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2026-02-13T01:54:30.366 INFO:tasks.workunit.client.0.trial112.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2026-02-13T01:54:30.366 INFO:tasks.workunit.client.0.trial112.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2026-02-13T01:54:30.366 INFO:tasks.workunit.client.0.trial112.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2026-02-13T01:54:30.366 INFO:tasks.workunit.client.0.trial112.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2026-02-13T01:54:30.366 INFO:tasks.workunit.client.0.trial112.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2026-02-13T01:54:30.366 INFO:tasks.workunit.client.0.trial112.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2026-02-13T01:54:30.366 INFO:tasks.workunit.client.0.trial112.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2026-02-13T01:54:30.366 INFO:tasks.workunit.client.0.trial112.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2026-02-13T01:54:30.366 INFO:tasks.workunit.client.0.trial112.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2026-02-13T01:54:30.366 INFO:tasks.workunit.client.0.trial112.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2026-02-13T01:54:30.366 INFO:tasks.workunit.client.0.trial112.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2026-02-13T01:54:30.366 INFO:tasks.workunit.client.0.trial112.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2026-02-13T01:54:30.367 INFO:tasks.workunit.client.0.trial112.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2026-02-13T01:54:30.367 INFO:tasks.workunit.client.0.trial112.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2026-02-13T01:54:30.367 INFO:tasks.workunit.client.0.trial112.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2026-02-13T01:54:30.367 INFO:tasks.workunit.client.0.trial112.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2026-02-13T01:54:30.367 INFO:tasks.workunit.client.0.trial112.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2026-02-13T01:54:30.367 INFO:tasks.workunit.client.0.trial112.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2026-02-13T01:54:30.367 INFO:tasks.workunit.client.0.trial112.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2026-02-13T01:54:30.367 INFO:tasks.workunit.client.0.trial112.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2026-02-13T01:54:30.367 INFO:tasks.workunit.client.0.trial112.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2026-02-13T01:54:30.367 INFO:tasks.workunit.client.0.trial112.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2026-02-13T01:54:30.367 INFO:tasks.workunit.client.0.trial112.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2026-02-13T01:54:30.367 INFO:tasks.workunit.client.0.trial112.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2026-02-13T01:54:30.367 INFO:tasks.workunit.client.0.trial112.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2026-02-13T01:54:30.368 INFO:tasks.workunit.client.0.trial112.stdout:00000220 72 65 18 00 00 00 05 00 00 00 fa 04 7f 72 1d 01 |re...........r..| 2026-02-13T01:54:30.368 INFO:tasks.workunit.client.0.trial112.stdout:00000230 45 6d bf fe 31 a0 4c d9 0f 71 00 00 00 00 00 00 |Em..1.L..q......| 2026-02-13T01:54:30.368 INFO:tasks.workunit.client.0.trial112.stdout:00000240 00 00 18 00 00 00 3f 84 8e 69 46 55 0f 23 01 00 |......?..iFU.#..| 2026-02-13T01:54:30.368 INFO:tasks.workunit.client.0.trial112.stdout:00000250 00 00 01 00 00 00 01 00 00 00 18 00 00 00 |..............| 2026-02-13T01:54:30.368 INFO:tasks.workunit.client.0.trial112.stdout:0000025e 2026-02-13T01:54:30.368 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:30.368 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2026-02-13T01:54:30.368 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:30.368 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2026-02-13T01:54:30.368 INFO:tasks.workunit.client.0.trial112.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2026-02-13T01:54:30.368 INFO:tasks.workunit.client.0.trial112.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2026-02-13T01:54:30.368 INFO:tasks.workunit.client.0.trial112.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-13T01:54:30.369 INFO:tasks.workunit.client.0.trial112.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2026-02-13T01:54:30.369 INFO:tasks.workunit.client.0.trial112.stdout:00000040 00 00 00 00 00 00 00 |.......| 2026-02-13T01:54:30.369 INFO:tasks.workunit.client.0.trial112.stdout:00000047 2026-02-13T01:54:30.369 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:30.369 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2026-02-13T01:54:30.369 INFO:tasks.workunit.client.0.trial112.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2026-02-13T01:54:30.369 INFO:tasks.workunit.client.0.trial112.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2026-02-13T01:54:30.369 INFO:tasks.workunit.client.0.trial112.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-13T01:54:30.369 INFO:tasks.workunit.client.0.trial112.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2026-02-13T01:54:30.369 INFO:tasks.workunit.client.0.trial112.stdout:0000003f 2026-02-13T01:54:30.369 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:30.369 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2026-02-13T01:54:30.370 INFO:tasks.workunit.client.0.trial112.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2026-02-13T01:54:30.370 INFO:tasks.workunit.client.0.trial112.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2026-02-13T01:54:30.370 INFO:tasks.workunit.client.0.trial112.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-13T01:54:30.370 INFO:tasks.workunit.client.0.trial112.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2026-02-13T01:54:30.370 INFO:tasks.workunit.client.0.trial112.stdout:00000040 00 00 00 00 00 00 00 |.......| 2026-02-13T01:54:30.370 INFO:tasks.workunit.client.0.trial112.stdout:00000047 2026-02-13T01:54:30.370 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:30.370 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2026-02-13T01:54:30.370 INFO:tasks.workunit.client.0.trial112.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-13T01:54:30.370 INFO:tasks.workunit.client.0.trial112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-13T01:54:30.370 INFO:tasks.workunit.client.0.trial112.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-13T01:54:30.370 INFO:tasks.workunit.client.0.trial112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-13T01:54:30.371 INFO:tasks.workunit.client.0.trial112.stdout:0000003b 2026-02-13T01:54:30.371 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:30.371 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2026-02-13T01:54:30.371 INFO:tasks.workunit.client.0.trial112.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-13T01:54:30.371 INFO:tasks.workunit.client.0.trial112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-13T01:54:30.371 INFO:tasks.workunit.client.0.trial112.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-13T01:54:30.371 INFO:tasks.workunit.client.0.trial112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-13T01:54:30.371 INFO:tasks.workunit.client.0.trial112.stdout:0000003b 2026-02-13T01:54:30.371 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:30.371 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-13T01:54:30.371 INFO:tasks.workunit.client.0.trial112.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-13T01:54:30.371 INFO:tasks.workunit.client.0.trial112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-13T01:54:30.372 INFO:tasks.workunit.client.0.trial112.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-13T01:54:30.372 INFO:tasks.workunit.client.0.trial112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-13T01:54:30.372 INFO:tasks.workunit.client.0.trial112.stdout:0000003b 2026-02-13T01:54:30.372 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:30.372 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2026-02-13T01:54:30.372 INFO:tasks.workunit.client.0.trial112.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-13T01:54:30.372 INFO:tasks.workunit.client.0.trial112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-13T01:54:30.372 INFO:tasks.workunit.client.0.trial112.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-13T01:54:30.372 INFO:tasks.workunit.client.0.trial112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-13T01:54:30.372 INFO:tasks.workunit.client.0.trial112.stdout:0000003b 2026-02-13T01:54:30.372 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:30.372 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2026-02-13T01:54:30.372 INFO:tasks.workunit.client.0.trial112.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-13T01:54:30.373 INFO:tasks.workunit.client.0.trial112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-13T01:54:30.373 INFO:tasks.workunit.client.0.trial112.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-13T01:54:30.373 INFO:tasks.workunit.client.0.trial112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-13T01:54:30.373 INFO:tasks.workunit.client.0.trial112.stdout:0000003b 2026-02-13T01:54:30.373 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:30.373 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2026-02-13T01:54:30.373 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:30.373 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2026-02-13T01:54:30.373 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:30.374 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.356+0000 7faabfa6cd40 1 bluefs umount 2026-02-13T01:54:30.374 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.356+0000 7faabfa6cd40 1 bdev(0x5600fab81c00 td/osd-mapper/0/block) close 2026-02-13T01:54:30.402 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.396+0000 7faabfa6cd40 1 freelist shutdown 2026-02-13T01:54:30.402 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:30.396+0000 7faabfa6cd40 1 bdev(0x5600fab81800 td/osd-mapper/0/block) close 2026-02-13T01:54:30.439 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2026-02-13T01:54:30.439 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2026-02-13T01:54:30.439 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2026-02-13T01:54:30.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-13T01:54:30.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2026-02-13T01:54:30.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2026-02-13T01:54:30.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2026-02-13T01:54:30.684 INFO:tasks.workunit.client.0.trial112.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-13T01:54:30.684 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2026-02-13T01:54:30.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.gNzCwOz0Uc_the_val 2026-02-13T01:54:30.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2026-02-13T01:54:30.685 INFO:tasks.workunit.client.0.trial112.stdout:Value dumped in: /tmp/tmp.gNzCwOz0Uc_the_val 2026-02-13T01:54:30.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.gNzCwOz0Uc_the_val' 2026-02-13T01:54:30.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.gNzCwOz0Uc_the_val 2026-02-13T01:54:30.823 INFO:tasks.workunit.client.0.trial112.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2026-02-13T01:54:30.906 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2026-02-13T01:54:30.906 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.gNzCwOz0Uc_the_val 2026-02-13T01:54:30.907 INFO:tasks.workunit.client.0.trial112.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2026-02-13T01:54:30.907 INFO:tasks.workunit.client.0.trial112.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2026-02-13T01:54:30.907 INFO:tasks.workunit.client.0.trial112.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2026-02-13T01:54:30.908 INFO:tasks.workunit.client.0.trial112.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2026-02-13T01:54:30.908 INFO:tasks.workunit.client.0.trial112.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2026-02-13T01:54:30.908 INFO:tasks.workunit.client.0.trial112.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2026-02-13T01:54:30.908 INFO:tasks.workunit.client.0.trial112.stdout:0000060 0000 0100 0000 0000 0000 0000 2026-02-13T01:54:30.908 INFO:tasks.workunit.client.0.trial112.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2026-02-13T01:54:30.908 INFO:tasks.workunit.client.0.trial112.stdout:0000073 2026-02-13T01:54:30.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2026-02-13T01:54:30.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-13T01:54:31.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.gNzCwOz0Uc_the_val 2026-02-13T01:54:31.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2026-02-13T01:54:31.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2026-02-13T01:54:31.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2026-02-13T01:54:31.374 INFO:tasks.workunit.client.0.trial112.stdout:corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1) 2026-02-13T01:54:31.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1)' 2026-02-13T01:54:31.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2026-02-13T01:54:31.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2026-02-13T01:54:31.386 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.376+0000 7fc219f40d40 1 bdev(0x55a2d0905800 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2026-02-13T01:54:31.386 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.376+0000 7fc219f40d40 1 bdev(0x55a2d0905800 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-13T01:54:31.387 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.380+0000 7fc219f40d40 1 bdev(0x55a2d0905800 td/osd-mapper/1/block) close 2026-02-13T01:54:31.430 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.424+0000 7fc219f40d40 1 bdev(0x55a2d0905800 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2026-02-13T01:54:31.431 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.424+0000 7fc219f40d40 1 bdev(0x55a2d0905800 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-13T01:54:31.432 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.424+0000 7fc219f40d40 1 bluestore(td/osd-mapper/1) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-13T01:54:31.432 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.424+0000 7fc219f40d40 1 bdev(0x55a2d0905c00 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2026-02-13T01:54:31.432 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.424+0000 7fc219f40d40 1 bdev(0x55a2d0905c00 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-13T01:54:31.432 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.424+0000 7fc219f40d40 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2026-02-13T01:54:31.432 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.424+0000 7fc219f40d40 1 bluefs mount 2026-02-13T01:54:31.432 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.424+0000 7fc219f40d40 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-13T01:54:31.434 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.428+0000 7fc219f40d40 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:31.435 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.428+0000 7fc219f40d40 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:31.435 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.428+0000 7fc219f40d40 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:31.435 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.428+0000 7fc219f40d40 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:31.435 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.428+0000 7fc219f40d40 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:31.435 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.428+0000 7fc219f40d40 1 bluefs mount shared_bdev_used = 0 2026-02-13T01:54:31.435 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.428+0000 7fc219f40d40 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-13T01:54:31.447 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.440+0000 7fc219f40d40 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-13T01:54:31.447 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.440+0000 7fc219f40d40 1 bluestore(td/osd-mapper/1) _open_super_meta old nid_max 2050 2026-02-13T01:54:31.448 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.440+0000 7fc219f40d40 1 bluestore(td/osd-mapper/1) _open_super_meta old blobid_max 20480 2026-02-13T01:54:31.448 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.440+0000 7fc219f40d40 1 bluestore(td/osd-mapper/1) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-13T01:54:31.448 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.440+0000 7fc219f40d40 1 bluestore(td/osd-mapper/1) _open_super_meta min_alloc_size 0x1000 2026-02-13T01:54:31.448 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.440+0000 7fc219f40d40 1 freelist init 2026-02-13T01:54:31.448 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.440+0000 7fc219f40d40 1 freelist _read_cfg 2026-02-13T01:54:31.448 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.440+0000 7fc219f40d40 1 bluestore(td/osd-mapper/1) _open_fm effective freelist_type = null, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2026-02-13T01:54:31.449 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.440+0000 7fc219f40d40 1 bluestore(td/osd-mapper/1) _init_alloc loaded 100 GiB in 16 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc8000, fragmentation 5.72206e-07 2026-02-13T01:54:31.449 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.440+0000 7fc219f40d40 1 bluefs umount 2026-02-13T01:54:31.449 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.440+0000 7fc219f40d40 1 bdev(0x55a2d0905c00 td/osd-mapper/1/block) close 2026-02-13T01:54:31.498 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.492+0000 7fc219f40d40 1 bdev(0x55a2d0905c00 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2026-02-13T01:54:31.499 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.492+0000 7fc219f40d40 1 bdev(0x55a2d0905c00 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-13T01:54:31.499 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.492+0000 7fc219f40d40 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2026-02-13T01:54:31.499 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.492+0000 7fc219f40d40 1 bluefs mount 2026-02-13T01:54:31.499 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.492+0000 7fc219f40d40 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-13T01:54:31.500 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.492+0000 7fc219f40d40 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:31.500 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.492+0000 7fc219f40d40 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:31.500 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.492+0000 7fc219f40d40 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:31.501 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.492+0000 7fc219f40d40 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:31.501 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.492+0000 7fc219f40d40 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:31.501 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.492+0000 7fc219f40d40 1 bluefs mount shared_bdev_used = 27656192 2026-02-13T01:54:31.501 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.492+0000 7fc219f40d40 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-13T01:54:31.514 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.504+0000 7fc219f40d40 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-13T01:54:31.514 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2026-02-13T01:54:31.515 INFO:tasks.workunit.client.0.trial112.stdout:00000000 0b 05 58 02 00 00 b7 9c f4 58 12 42 4c c6 8f 9e |..X......X.BL...| 2026-02-13T01:54:31.515 INFO:tasks.workunit.client.0.trial112.stdout:00000010 48 1a 9a ef bd ea 01 00 00 00 18 00 00 00 00 00 |H...............| 2026-02-13T01:54:31.515 INFO:tasks.workunit.client.0.trial112.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2026-02-13T01:54:31.515 INFO:tasks.workunit.client.0.trial112.stdout:* 2026-02-13T01:54:31.515 INFO:tasks.workunit.client.0.trial112.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2026-02-13T01:54:31.515 INFO:tasks.workunit.client.0.trial112.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2026-02-13T01:54:31.515 INFO:tasks.workunit.client.0.trial112.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2026-02-13T01:54:31.515 INFO:tasks.workunit.client.0.trial112.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2026-02-13T01:54:31.515 INFO:tasks.workunit.client.0.trial112.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2026-02-13T01:54:31.515 INFO:tasks.workunit.client.0.trial112.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2026-02-13T01:54:31.516 INFO:tasks.workunit.client.0.trial112.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2026-02-13T01:54:31.516 INFO:tasks.workunit.client.0.trial112.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2026-02-13T01:54:31.516 INFO:tasks.workunit.client.0.trial112.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2026-02-13T01:54:31.516 INFO:tasks.workunit.client.0.trial112.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2026-02-13T01:54:31.516 INFO:tasks.workunit.client.0.trial112.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2026-02-13T01:54:31.516 INFO:tasks.workunit.client.0.trial112.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2026-02-13T01:54:31.516 INFO:tasks.workunit.client.0.trial112.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2026-02-13T01:54:31.516 INFO:tasks.workunit.client.0.trial112.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2026-02-13T01:54:31.516 INFO:tasks.workunit.client.0.trial112.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2026-02-13T01:54:31.516 INFO:tasks.workunit.client.0.trial112.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2026-02-13T01:54:31.516 INFO:tasks.workunit.client.0.trial112.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2026-02-13T01:54:31.516 INFO:tasks.workunit.client.0.trial112.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2026-02-13T01:54:31.517 INFO:tasks.workunit.client.0.trial112.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2026-02-13T01:54:31.517 INFO:tasks.workunit.client.0.trial112.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2026-02-13T01:54:31.517 INFO:tasks.workunit.client.0.trial112.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2026-02-13T01:54:31.517 INFO:tasks.workunit.client.0.trial112.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2026-02-13T01:54:31.517 INFO:tasks.workunit.client.0.trial112.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2026-02-13T01:54:31.517 INFO:tasks.workunit.client.0.trial112.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2026-02-13T01:54:31.517 INFO:tasks.workunit.client.0.trial112.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2026-02-13T01:54:31.517 INFO:tasks.workunit.client.0.trial112.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2026-02-13T01:54:31.517 INFO:tasks.workunit.client.0.trial112.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2026-02-13T01:54:31.517 INFO:tasks.workunit.client.0.trial112.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2026-02-13T01:54:31.517 INFO:tasks.workunit.client.0.trial112.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2026-02-13T01:54:31.517 INFO:tasks.workunit.client.0.trial112.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2026-02-13T01:54:31.517 INFO:tasks.workunit.client.0.trial112.stdout:00000220 72 65 18 00 00 00 09 00 00 00 58 ef 68 40 d5 67 |re........X.h@.g| 2026-02-13T01:54:31.518 INFO:tasks.workunit.client.0.trial112.stdout:00000230 48 3d 9a ea 75 b7 41 86 09 7e 00 00 00 00 00 00 |H=..u.A..~......| 2026-02-13T01:54:31.518 INFO:tasks.workunit.client.0.trial112.stdout:00000240 00 00 18 00 00 00 42 84 8e 69 1f 25 6b 24 01 00 |......B..i.%k$..| 2026-02-13T01:54:31.518 INFO:tasks.workunit.client.0.trial112.stdout:00000250 00 00 01 00 00 00 01 00 00 00 18 00 00 00 |..............| 2026-02-13T01:54:31.518 INFO:tasks.workunit.client.0.trial112.stdout:0000025e 2026-02-13T01:54:31.518 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:31.518 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2026-02-13T01:54:31.518 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:31.518 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2026-02-13T01:54:31.518 INFO:tasks.workunit.client.0.trial112.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2026-02-13T01:54:31.518 INFO:tasks.workunit.client.0.trial112.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2026-02-13T01:54:31.518 INFO:tasks.workunit.client.0.trial112.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-13T01:54:31.518 INFO:tasks.workunit.client.0.trial112.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2026-02-13T01:54:31.519 INFO:tasks.workunit.client.0.trial112.stdout:00000040 00 00 00 00 00 00 00 |.......| 2026-02-13T01:54:31.519 INFO:tasks.workunit.client.0.trial112.stdout:00000047 2026-02-13T01:54:31.519 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:31.519 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2026-02-13T01:54:31.519 INFO:tasks.workunit.client.0.trial112.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2026-02-13T01:54:31.519 INFO:tasks.workunit.client.0.trial112.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2026-02-13T01:54:31.519 INFO:tasks.workunit.client.0.trial112.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-13T01:54:31.519 INFO:tasks.workunit.client.0.trial112.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2026-02-13T01:54:31.519 INFO:tasks.workunit.client.0.trial112.stdout:0000003f 2026-02-13T01:54:31.519 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:31.519 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2026-02-13T01:54:31.519 INFO:tasks.workunit.client.0.trial112.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2026-02-13T01:54:31.520 INFO:tasks.workunit.client.0.trial112.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2026-02-13T01:54:31.520 INFO:tasks.workunit.client.0.trial112.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-13T01:54:31.520 INFO:tasks.workunit.client.0.trial112.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2026-02-13T01:54:31.520 INFO:tasks.workunit.client.0.trial112.stdout:00000040 00 00 00 00 00 00 00 |.......| 2026-02-13T01:54:31.520 INFO:tasks.workunit.client.0.trial112.stdout:00000047 2026-02-13T01:54:31.520 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:31.520 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2026-02-13T01:54:31.520 INFO:tasks.workunit.client.0.trial112.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-13T01:54:31.520 INFO:tasks.workunit.client.0.trial112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-13T01:54:31.520 INFO:tasks.workunit.client.0.trial112.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-13T01:54:31.520 INFO:tasks.workunit.client.0.trial112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-13T01:54:31.520 INFO:tasks.workunit.client.0.trial112.stdout:0000003b 2026-02-13T01:54:31.521 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:31.521 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2026-02-13T01:54:31.521 INFO:tasks.workunit.client.0.trial112.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-13T01:54:31.521 INFO:tasks.workunit.client.0.trial112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-13T01:54:31.521 INFO:tasks.workunit.client.0.trial112.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-13T01:54:31.521 INFO:tasks.workunit.client.0.trial112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-13T01:54:31.521 INFO:tasks.workunit.client.0.trial112.stdout:0000003b 2026-02-13T01:54:31.521 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:31.521 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-13T01:54:31.521 INFO:tasks.workunit.client.0.trial112.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-13T01:54:31.521 INFO:tasks.workunit.client.0.trial112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-13T01:54:31.521 INFO:tasks.workunit.client.0.trial112.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-13T01:54:31.522 INFO:tasks.workunit.client.0.trial112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-13T01:54:31.522 INFO:tasks.workunit.client.0.trial112.stdout:0000003b 2026-02-13T01:54:31.522 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:31.522 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2026-02-13T01:54:31.522 INFO:tasks.workunit.client.0.trial112.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-13T01:54:31.522 INFO:tasks.workunit.client.0.trial112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-13T01:54:31.522 INFO:tasks.workunit.client.0.trial112.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-13T01:54:31.522 INFO:tasks.workunit.client.0.trial112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-13T01:54:31.522 INFO:tasks.workunit.client.0.trial112.stdout:0000003b 2026-02-13T01:54:31.522 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:31.522 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2026-02-13T01:54:31.522 INFO:tasks.workunit.client.0.trial112.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-13T01:54:31.522 INFO:tasks.workunit.client.0.trial112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-13T01:54:31.523 INFO:tasks.workunit.client.0.trial112.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-13T01:54:31.523 INFO:tasks.workunit.client.0.trial112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-13T01:54:31.523 INFO:tasks.workunit.client.0.trial112.stdout:0000003b 2026-02-13T01:54:31.523 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:31.523 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2026-02-13T01:54:31.523 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:31.523 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2026-02-13T01:54:31.523 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:31.524 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.504+0000 7fc219f40d40 1 bluefs umount 2026-02-13T01:54:31.524 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.504+0000 7fc219f40d40 1 bdev(0x55a2d0905c00 td/osd-mapper/1/block) close 2026-02-13T01:54:31.554 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.548+0000 7fc219f40d40 1 freelist shutdown 2026-02-13T01:54:31.554 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:31.548+0000 7fc219f40d40 1 bdev(0x55a2d0905800 td/osd-mapper/1/block) close 2026-02-13T01:54:31.598 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2026-02-13T01:54:31.599 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2026-02-13T01:54:31.599 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2026-02-13T01:54:31.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-13T01:54:31.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2026-02-13T01:54:31.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2026-02-13T01:54:31.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2026-02-13T01:54:31.820 INFO:tasks.workunit.client.0.trial112.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-13T01:54:31.820 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2026-02-13T01:54:31.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.W4gXXigIzd_the_val 2026-02-13T01:54:31.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2026-02-13T01:54:31.821 INFO:tasks.workunit.client.0.trial112.stdout:Value dumped in: /tmp/tmp.W4gXXigIzd_the_val 2026-02-13T01:54:31.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.W4gXXigIzd_the_val' 2026-02-13T01:54:31.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.W4gXXigIzd_the_val 2026-02-13T01:54:31.968 INFO:tasks.workunit.client.0.trial112.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2026-02-13T01:54:32.054 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2026-02-13T01:54:32.054 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.W4gXXigIzd_the_val 2026-02-13T01:54:32.054 INFO:tasks.workunit.client.0.trial112.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2026-02-13T01:54:32.055 INFO:tasks.workunit.client.0.trial112.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2026-02-13T01:54:32.055 INFO:tasks.workunit.client.0.trial112.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2026-02-13T01:54:32.055 INFO:tasks.workunit.client.0.trial112.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2026-02-13T01:54:32.055 INFO:tasks.workunit.client.0.trial112.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2026-02-13T01:54:32.055 INFO:tasks.workunit.client.0.trial112.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2026-02-13T01:54:32.055 INFO:tasks.workunit.client.0.trial112.stdout:0000060 0000 0100 0000 0000 0000 0000 2026-02-13T01:54:32.055 INFO:tasks.workunit.client.0.trial112.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2026-02-13T01:54:32.055 INFO:tasks.workunit.client.0.trial112.stdout:0000073 2026-02-13T01:54:32.056 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2026-02-13T01:54:32.056 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-13T01:54:32.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.W4gXXigIzd_the_val 2026-02-13T01:54:32.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2026-02-13T01:54:32.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2026-02-13T01:54:32.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/2 2026-02-13T01:54:32.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2)' 2026-02-13T01:54:32.518 INFO:tasks.workunit.client.0.trial112.stdout:corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2) 2026-02-13T01:54:32.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2026-02-13T01:54:32.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2026-02-13T01:54:32.529 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.520+0000 7f3e9113fd40 1 bdev(0x5619db113800 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2026-02-13T01:54:32.530 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.520+0000 7f3e9113fd40 1 bdev(0x5619db113800 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-13T01:54:32.530 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.520+0000 7f3e9113fd40 1 bdev(0x5619db113800 td/osd-mapper/2/block) close 2026-02-13T01:54:32.574 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.568+0000 7f3e9113fd40 1 bdev(0x5619db113800 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2026-02-13T01:54:32.574 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.568+0000 7f3e9113fd40 1 bdev(0x5619db113800 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-13T01:54:32.575 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.568+0000 7f3e9113fd40 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-13T01:54:32.575 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.568+0000 7f3e9113fd40 1 bdev(0x5619db113c00 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2026-02-13T01:54:32.575 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.568+0000 7f3e9113fd40 1 bdev(0x5619db113c00 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-13T01:54:32.576 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.568+0000 7f3e9113fd40 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2026-02-13T01:54:32.576 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.568+0000 7f3e9113fd40 1 bluefs mount 2026-02-13T01:54:32.576 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.568+0000 7f3e9113fd40 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-13T01:54:32.578 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.568+0000 7f3e9113fd40 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:32.578 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.568+0000 7f3e9113fd40 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:32.578 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.568+0000 7f3e9113fd40 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:32.578 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.568+0000 7f3e9113fd40 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:32.578 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.568+0000 7f3e9113fd40 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:32.579 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.568+0000 7f3e9113fd40 1 bluefs mount shared_bdev_used = 0 2026-02-13T01:54:32.579 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.568+0000 7f3e9113fd40 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-13T01:54:32.590 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.580+0000 7f3e9113fd40 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-13T01:54:32.590 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.580+0000 7f3e9113fd40 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2051 2026-02-13T01:54:32.590 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.580+0000 7f3e9113fd40 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2026-02-13T01:54:32.590 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.580+0000 7f3e9113fd40 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-13T01:54:32.591 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.580+0000 7f3e9113fd40 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2026-02-13T01:54:32.591 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.584+0000 7f3e9113fd40 1 freelist init 2026-02-13T01:54:32.591 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.584+0000 7f3e9113fd40 1 freelist _read_cfg 2026-02-13T01:54:32.591 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.584+0000 7f3e9113fd40 1 bluestore(td/osd-mapper/2) _open_fm effective freelist_type = null, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2026-02-13T01:54:32.592 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.584+0000 7f3e9113fd40 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 12 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc6000, fragmentation 4.19618e-07 2026-02-13T01:54:32.592 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.584+0000 7f3e9113fd40 1 bluefs umount 2026-02-13T01:54:32.592 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.584+0000 7f3e9113fd40 1 bdev(0x5619db113c00 td/osd-mapper/2/block) close 2026-02-13T01:54:32.634 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.628+0000 7f3e9113fd40 1 bdev(0x5619db113c00 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2026-02-13T01:54:32.635 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.628+0000 7f3e9113fd40 1 bdev(0x5619db113c00 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-13T01:54:32.635 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.628+0000 7f3e9113fd40 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2026-02-13T01:54:32.635 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.628+0000 7f3e9113fd40 1 bluefs mount 2026-02-13T01:54:32.635 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.628+0000 7f3e9113fd40 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-13T01:54:32.636 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.628+0000 7f3e9113fd40 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:32.636 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.628+0000 7f3e9113fd40 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:32.637 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.628+0000 7f3e9113fd40 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:32.637 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.628+0000 7f3e9113fd40 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:32.637 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.628+0000 7f3e9113fd40 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-13T01:54:32.637 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.628+0000 7f3e9113fd40 1 bluefs mount shared_bdev_used = 27656192 2026-02-13T01:54:32.637 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.628+0000 7f3e9113fd40 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-13T01:54:32.648 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.640+0000 7f3e9113fd40 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-13T01:54:32.649 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2026-02-13T01:54:32.649 INFO:tasks.workunit.client.0.trial112.stdout:00000000 0b 05 58 02 00 00 b7 9c f4 58 12 42 4c c6 8f 9e |..X......X.BL...| 2026-02-13T01:54:32.649 INFO:tasks.workunit.client.0.trial112.stdout:00000010 48 1a 9a ef bd ea 02 00 00 00 19 00 00 00 00 00 |H...............| 2026-02-13T01:54:32.649 INFO:tasks.workunit.client.0.trial112.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2026-02-13T01:54:32.649 INFO:tasks.workunit.client.0.trial112.stdout:* 2026-02-13T01:54:32.649 INFO:tasks.workunit.client.0.trial112.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2026-02-13T01:54:32.649 INFO:tasks.workunit.client.0.trial112.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2026-02-13T01:54:32.649 INFO:tasks.workunit.client.0.trial112.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2026-02-13T01:54:32.649 INFO:tasks.workunit.client.0.trial112.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2026-02-13T01:54:32.649 INFO:tasks.workunit.client.0.trial112.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2026-02-13T01:54:32.650 INFO:tasks.workunit.client.0.trial112.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2026-02-13T01:54:32.650 INFO:tasks.workunit.client.0.trial112.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2026-02-13T01:54:32.650 INFO:tasks.workunit.client.0.trial112.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2026-02-13T01:54:32.650 INFO:tasks.workunit.client.0.trial112.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2026-02-13T01:54:32.650 INFO:tasks.workunit.client.0.trial112.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2026-02-13T01:54:32.650 INFO:tasks.workunit.client.0.trial112.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2026-02-13T01:54:32.650 INFO:tasks.workunit.client.0.trial112.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2026-02-13T01:54:32.650 INFO:tasks.workunit.client.0.trial112.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2026-02-13T01:54:32.650 INFO:tasks.workunit.client.0.trial112.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2026-02-13T01:54:32.650 INFO:tasks.workunit.client.0.trial112.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2026-02-13T01:54:32.650 INFO:tasks.workunit.client.0.trial112.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2026-02-13T01:54:32.650 INFO:tasks.workunit.client.0.trial112.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2026-02-13T01:54:32.651 INFO:tasks.workunit.client.0.trial112.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2026-02-13T01:54:32.651 INFO:tasks.workunit.client.0.trial112.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2026-02-13T01:54:32.651 INFO:tasks.workunit.client.0.trial112.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2026-02-13T01:54:32.651 INFO:tasks.workunit.client.0.trial112.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2026-02-13T01:54:32.651 INFO:tasks.workunit.client.0.trial112.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2026-02-13T01:54:32.651 INFO:tasks.workunit.client.0.trial112.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2026-02-13T01:54:32.651 INFO:tasks.workunit.client.0.trial112.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2026-02-13T01:54:32.651 INFO:tasks.workunit.client.0.trial112.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2026-02-13T01:54:32.651 INFO:tasks.workunit.client.0.trial112.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2026-02-13T01:54:32.651 INFO:tasks.workunit.client.0.trial112.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2026-02-13T01:54:32.651 INFO:tasks.workunit.client.0.trial112.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2026-02-13T01:54:32.651 INFO:tasks.workunit.client.0.trial112.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2026-02-13T01:54:32.651 INFO:tasks.workunit.client.0.trial112.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2026-02-13T01:54:32.652 INFO:tasks.workunit.client.0.trial112.stdout:00000220 72 65 19 00 00 00 0d 00 00 00 33 94 d7 31 6c 3f |re........3..1l?| 2026-02-13T01:54:32.652 INFO:tasks.workunit.client.0.trial112.stdout:00000230 4a db aa 2d 10 1b 26 c4 10 6b 00 00 00 00 00 00 |J..-..&..k......| 2026-02-13T01:54:32.652 INFO:tasks.workunit.client.0.trial112.stdout:00000240 00 00 19 00 00 00 46 84 8e 69 81 17 58 10 01 00 |......F..i..X...| 2026-02-13T01:54:32.652 INFO:tasks.workunit.client.0.trial112.stdout:00000250 00 00 01 00 00 00 01 00 00 00 19 00 00 00 |..............| 2026-02-13T01:54:32.652 INFO:tasks.workunit.client.0.trial112.stdout:0000025e 2026-02-13T01:54:32.652 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:32.652 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2026-02-13T01:54:32.652 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:32.652 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2026-02-13T01:54:32.652 INFO:tasks.workunit.client.0.trial112.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2026-02-13T01:54:32.652 INFO:tasks.workunit.client.0.trial112.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2026-02-13T01:54:32.652 INFO:tasks.workunit.client.0.trial112.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-13T01:54:32.653 INFO:tasks.workunit.client.0.trial112.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2026-02-13T01:54:32.653 INFO:tasks.workunit.client.0.trial112.stdout:00000040 00 00 00 00 00 00 00 |.......| 2026-02-13T01:54:32.653 INFO:tasks.workunit.client.0.trial112.stdout:00000047 2026-02-13T01:54:32.653 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:32.653 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2026-02-13T01:54:32.653 INFO:tasks.workunit.client.0.trial112.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2026-02-13T01:54:32.653 INFO:tasks.workunit.client.0.trial112.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2026-02-13T01:54:32.653 INFO:tasks.workunit.client.0.trial112.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-13T01:54:32.653 INFO:tasks.workunit.client.0.trial112.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2026-02-13T01:54:32.653 INFO:tasks.workunit.client.0.trial112.stdout:0000003f 2026-02-13T01:54:32.653 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:32.653 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2026-02-13T01:54:32.654 INFO:tasks.workunit.client.0.trial112.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2026-02-13T01:54:32.654 INFO:tasks.workunit.client.0.trial112.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2026-02-13T01:54:32.654 INFO:tasks.workunit.client.0.trial112.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-13T01:54:32.654 INFO:tasks.workunit.client.0.trial112.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2026-02-13T01:54:32.654 INFO:tasks.workunit.client.0.trial112.stdout:00000040 00 00 00 00 00 00 00 |.......| 2026-02-13T01:54:32.654 INFO:tasks.workunit.client.0.trial112.stdout:00000047 2026-02-13T01:54:32.654 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:32.654 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2026-02-13T01:54:32.654 INFO:tasks.workunit.client.0.trial112.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-13T01:54:32.654 INFO:tasks.workunit.client.0.trial112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-13T01:54:32.654 INFO:tasks.workunit.client.0.trial112.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-13T01:54:32.654 INFO:tasks.workunit.client.0.trial112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-13T01:54:32.654 INFO:tasks.workunit.client.0.trial112.stdout:0000003b 2026-02-13T01:54:32.655 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:32.655 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2026-02-13T01:54:32.655 INFO:tasks.workunit.client.0.trial112.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-13T01:54:32.655 INFO:tasks.workunit.client.0.trial112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-13T01:54:32.655 INFO:tasks.workunit.client.0.trial112.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-13T01:54:32.655 INFO:tasks.workunit.client.0.trial112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-13T01:54:32.655 INFO:tasks.workunit.client.0.trial112.stdout:0000003b 2026-02-13T01:54:32.655 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:32.655 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-13T01:54:32.655 INFO:tasks.workunit.client.0.trial112.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-13T01:54:32.655 INFO:tasks.workunit.client.0.trial112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-13T01:54:32.655 INFO:tasks.workunit.client.0.trial112.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-13T01:54:32.656 INFO:tasks.workunit.client.0.trial112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-13T01:54:32.656 INFO:tasks.workunit.client.0.trial112.stdout:0000003b 2026-02-13T01:54:32.656 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:32.656 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2026-02-13T01:54:32.656 INFO:tasks.workunit.client.0.trial112.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-13T01:54:32.656 INFO:tasks.workunit.client.0.trial112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-13T01:54:32.656 INFO:tasks.workunit.client.0.trial112.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-13T01:54:32.656 INFO:tasks.workunit.client.0.trial112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-13T01:54:32.656 INFO:tasks.workunit.client.0.trial112.stdout:0000003b 2026-02-13T01:54:32.656 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:32.656 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2026-02-13T01:54:32.656 INFO:tasks.workunit.client.0.trial112.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-13T01:54:32.656 INFO:tasks.workunit.client.0.trial112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-13T01:54:32.657 INFO:tasks.workunit.client.0.trial112.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-13T01:54:32.657 INFO:tasks.workunit.client.0.trial112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-13T01:54:32.657 INFO:tasks.workunit.client.0.trial112.stdout:0000003b 2026-02-13T01:54:32.657 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:32.657 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2026-02-13T01:54:32.657 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:32.657 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2026-02-13T01:54:32.657 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:32.658 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.640+0000 7f3e9113fd40 1 bluefs umount 2026-02-13T01:54:32.658 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.640+0000 7f3e9113fd40 1 bdev(0x5619db113c00 td/osd-mapper/2/block) close 2026-02-13T01:54:32.694 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.688+0000 7f3e9113fd40 1 freelist shutdown 2026-02-13T01:54:32.694 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:32.688+0000 7f3e9113fd40 1 bdev(0x5619db113800 td/osd-mapper/2/block) close 2026-02-13T01:54:32.746 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2026-02-13T01:54:32.746 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2026-02-13T01:54:32.747 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2026-02-13T01:54:32.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-13T01:54:32.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2026-02-13T01:54:32.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2026-02-13T01:54:32.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2026-02-13T01:54:32.980 INFO:tasks.workunit.client.0.trial112.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-13T01:54:32.980 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2026-02-13T01:54:32.981 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.Qjrhqnxtwx_the_val 2026-02-13T01:54:32.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2026-02-13T01:54:32.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.Qjrhqnxtwx_the_val' 2026-02-13T01:54:32.982 INFO:tasks.workunit.client.0.trial112.stdout:Value dumped in: /tmp/tmp.Qjrhqnxtwx_the_val 2026-02-13T01:54:32.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.Qjrhqnxtwx_the_val 2026-02-13T01:54:33.124 INFO:tasks.workunit.client.0.trial112.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2026-02-13T01:54:33.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2026-02-13T01:54:33.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.Qjrhqnxtwx_the_val 2026-02-13T01:54:33.215 INFO:tasks.workunit.client.0.trial112.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2026-02-13T01:54:33.215 INFO:tasks.workunit.client.0.trial112.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2026-02-13T01:54:33.215 INFO:tasks.workunit.client.0.trial112.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2026-02-13T01:54:33.215 INFO:tasks.workunit.client.0.trial112.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2026-02-13T01:54:33.215 INFO:tasks.workunit.client.0.trial112.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2026-02-13T01:54:33.215 INFO:tasks.workunit.client.0.trial112.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2026-02-13T01:54:33.215 INFO:tasks.workunit.client.0.trial112.stdout:0000060 0000 0100 0000 0000 0000 0000 2026-02-13T01:54:33.215 INFO:tasks.workunit.client.0.trial112.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2026-02-13T01:54:33.216 INFO:tasks.workunit.client.0.trial112.stdout:0000073 2026-02-13T01:54:33.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2026-02-13T01:54:33.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-13T01:54:33.441 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.Qjrhqnxtwx_the_val 2026-02-13T01:54:33.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2026-02-13T01:54:33.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:129: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3 ' 2026-02-13T01:54:33.651 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: echo --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3 2026-02-13T01:54:33.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3' 2026-02-13T01:54:33.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2026-02-13T01:54:33.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: echo 'Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3/' 2026-02-13T01:54:33.652 INFO:tasks.workunit.client.0.trial112.stdout:Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3/ 2026-02-13T01:54:33.652 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: expr 3 - 1 2026-02-13T01:54:33.653 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: seq 0 2 2026-02-13T01:54:33.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2026-02-13T01:54:33.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=b79cf458-1242-4cc6-8f9e-481a9aefbdea --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3' 2026-02-13T01:54:33.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 0 2026-02-13T01:54:33.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-mapper 2026-02-13T01:54:33.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T01:54:33.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T01:54:33.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T01:54:33.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-mapper/0 2026-02-13T01:54:33.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=b79cf458-1242-4cc6-8f9e-481a9aefbdea --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3' 2026-02-13T01:54:33.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T01:54:33.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T01:54:33.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T01:54:33.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/0' 2026-02-13T01:54:33.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/0/journal' 2026-02-13T01:54:33.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T01:54:33.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T01:54:33.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2026-02-13T01:54:33.658 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T01:54:33.658 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:54:33.658 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:54:33.658 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:54:33.658 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:54:33.658 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7480 2026-02-13T01:54:33.658 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.7480/$cluster-$name.asok' 2026-02-13T01:54:33.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.7480/$cluster-$name.asok' 2026-02-13T01:54:33.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T01:54:33.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2026-02-13T01:54:33.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2026-02-13T01:54:33.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T01:54:33.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T01:54:33.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T01:54:33.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T01:54:33.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T01:54:33.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T01:54:33.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-mapper/0 2026-02-13T01:54:33.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T01:54:33.660 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T01:54:33.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=b79cf458-1242-4cc6-8f9e-481a9aefbdea --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/0 --osd-journal=td/osd-mapper/0/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.7480/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T01:54:33.661 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-mapper/0/whoami 2026-02-13T01:54:33.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T01:54:33.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T01:54:33.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T01:54:33.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T01:54:33.685 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:33.676+0000 7fd080095a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:54:33.687 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:33.680+0000 7fd080095a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:54:33.688 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:33.680+0000 7fd080095a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:54:33.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T01:54:33.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T01:54:33.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T01:54:33.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T01:54:33.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T01:54:33.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:54:33.833 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T01:54:33.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T01:54:33.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:54:33.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T01:54:34.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:54:34.108 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:34.100+0000 7fd080095a40 -1 Falling back to public interface 2026-02-13T01:54:34.723 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:34.716+0000 7fd080095a40 -1 osd.0 24 log_to_monitors true 2026-02-13T01:54:35.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:54:35.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:54:35.009 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T01:54:35.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T01:54:35.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:54:35.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T01:54:35.182 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 28 up_thru 28 down_at 25 last_clean_interval [5,24) [v2:127.0.0.1:6800/3971845452,v1:127.0.0.1:6801/3971845452] [v2:127.0.0.1:6802/3971845452,v1:127.0.0.1:6803/3971845452] exists,up fa047f72-1d01-456d-bffe-31a04cd90f71 2026-02-13T01:54:35.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T01:54:35.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T01:54:35.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T01:54:35.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2026-02-13T01:54:35.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=b79cf458-1242-4cc6-8f9e-481a9aefbdea --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3' 2026-02-13T01:54:35.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 1 2026-02-13T01:54:35.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-mapper 2026-02-13T01:54:35.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T01:54:35.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T01:54:35.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T01:54:35.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-mapper/1 2026-02-13T01:54:35.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=b79cf458-1242-4cc6-8f9e-481a9aefbdea --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3' 2026-02-13T01:54:35.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T01:54:35.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T01:54:35.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T01:54:35.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/1' 2026-02-13T01:54:35.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/1/journal' 2026-02-13T01:54:35.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T01:54:35.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T01:54:35.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2026-02-13T01:54:35.188 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T01:54:35.188 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:54:35.188 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:54:35.188 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:54:35.188 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:54:35.188 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7480 2026-02-13T01:54:35.188 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.7480/$cluster-$name.asok' 2026-02-13T01:54:35.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.7480/$cluster-$name.asok' 2026-02-13T01:54:35.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T01:54:35.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2026-02-13T01:54:35.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2026-02-13T01:54:35.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T01:54:35.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T01:54:35.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T01:54:35.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T01:54:35.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T01:54:35.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T01:54:35.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-mapper/1 2026-02-13T01:54:35.189 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T01:54:35.190 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T01:54:35.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=b79cf458-1242-4cc6-8f9e-481a9aefbdea --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/1 --osd-journal=td/osd-mapper/1/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.7480/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T01:54:35.191 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-mapper/1/whoami 2026-02-13T01:54:35.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T01:54:35.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T01:54:35.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T01:54:35.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T01:54:35.217 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:35.208+0000 7f084499fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:54:35.218 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:35.208+0000 7f084499fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:54:35.219 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:35.212+0000 7f084499fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:54:35.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T01:54:35.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T01:54:35.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T01:54:35.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T01:54:35.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T01:54:35.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:54:35.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T01:54:35.365 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T01:54:35.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:54:35.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T01:54:35.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:54:35.564 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:35.556+0000 7f084499fa40 -1 Falling back to public interface 2026-02-13T01:54:36.142 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:36.132+0000 7f084499fa40 -1 osd.1 24 log_to_monitors true 2026-02-13T01:54:36.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:54:36.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:54:36.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T01:54:36.544 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T01:54:36.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:54:36.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T01:54:36.711 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 31 up_thru 31 down_at 25 last_clean_interval [9,24) [v2:127.0.0.1:6810/1050678171,v1:127.0.0.1:6811/1050678171] [v2:127.0.0.1:6812/1050678171,v1:127.0.0.1:6813/1050678171] exists,up 58ef6840-d567-483d-9aea-75b74186097e 2026-02-13T01:54:36.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T01:54:36.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T01:54:36.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T01:54:36.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2026-02-13T01:54:36.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=b79cf458-1242-4cc6-8f9e-481a9aefbdea --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3' 2026-02-13T01:54:36.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 2 2026-02-13T01:54:36.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-mapper 2026-02-13T01:54:36.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T01:54:36.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-13T01:54:36.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T01:54:36.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-mapper/2 2026-02-13T01:54:36.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=b79cf458-1242-4cc6-8f9e-481a9aefbdea --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3' 2026-02-13T01:54:36.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T01:54:36.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T01:54:36.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T01:54:36.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/2' 2026-02-13T01:54:36.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/2/journal' 2026-02-13T01:54:36.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T01:54:36.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T01:54:36.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2026-02-13T01:54:36.716 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T01:54:36.716 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:54:36.716 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:54:36.716 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:54:36.716 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:54:36.716 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7480 2026-02-13T01:54:36.717 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.7480/$cluster-$name.asok' 2026-02-13T01:54:36.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.7480/$cluster-$name.asok' 2026-02-13T01:54:36.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T01:54:36.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2026-02-13T01:54:36.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2026-02-13T01:54:36.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T01:54:36.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T01:54:36.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T01:54:36.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T01:54:36.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T01:54:36.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T01:54:36.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-mapper/2 2026-02-13T01:54:36.718 INFO:tasks.workunit.client.0.trial112.stdout:start osd.2 2026-02-13T01:54:36.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-13T01:54:36.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=b79cf458-1242-4cc6-8f9e-481a9aefbdea --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/2 --osd-journal=td/osd-mapper/2/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.7480/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T01:54:36.719 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-mapper/2/whoami 2026-02-13T01:54:36.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-13T01:54:36.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T01:54:36.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T01:54:36.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T01:54:36.740 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:36.732+0000 7f303f3bba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:54:36.742 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:36.732+0000 7f303f3bba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:54:36.743 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:36.736+0000 7f303f3bba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:54:36.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-13T01:54:36.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T01:54:36.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T01:54:36.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T01:54:36.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T01:54:36.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:54:36.888 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T01:54:36.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T01:54:36.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:54:36.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T01:54:37.064 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:54:37.179 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:37.172+0000 7f303f3bba40 -1 Falling back to public interface 2026-02-13T01:54:37.813 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:37.804+0000 7f303f3bba40 -1 osd.2 25 log_to_monitors true 2026-02-13T01:54:38.065 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:54:38.065 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:54:38.065 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T01:54:38.066 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T01:54:38.066 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:54:38.066 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T01:54:38.240 INFO:tasks.workunit.client.0.trial112.stdout:osd.2 up in weight 1 up_from 35 up_thru 35 down_at 25 last_clean_interval [13,25) [v2:127.0.0.1:6818/125156958,v1:127.0.0.1:6819/125156958] [v2:127.0.0.1:6820/125156958,v1:127.0.0.1:6821/125156958] exists,up 3394d731-6c3f-4adb-aa2d-101b26c4106b 2026-02-13T01:54:38.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T01:54:38.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T01:54:38.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T01:54:38.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:136: TEST_truncated_sna_record: sleep 1 2026-02-13T01:54:39.243 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: expr 3 - 1 2026-02-13T01:54:39.245 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: seq 0 2 2026-02-13T01:54:39.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2026-02-13T01:54:39.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.0 version 2026-02-13T01:54:39.310 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T01:54:39.311 INFO:tasks.workunit.client.0.trial112.stdout: "version": "20.2.0-636-g3b40640f", 2026-02-13T01:54:39.311 INFO:tasks.workunit.client.0.trial112.stdout: "release": "tentacle", 2026-02-13T01:54:39.311 INFO:tasks.workunit.client.0.trial112.stdout: "release_type": "stable" 2026-02-13T01:54:39.311 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T01:54:39.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2026-02-13T01:54:39.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.1 version 2026-02-13T01:54:39.383 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T01:54:39.383 INFO:tasks.workunit.client.0.trial112.stdout: "version": "20.2.0-636-g3b40640f", 2026-02-13T01:54:39.383 INFO:tasks.workunit.client.0.trial112.stdout: "release": "tentacle", 2026-02-13T01:54:39.383 INFO:tasks.workunit.client.0.trial112.stdout: "release_type": "stable" 2026-02-13T01:54:39.383 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T01:54:39.393 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2026-02-13T01:54:39.393 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.2 version 2026-02-13T01:54:39.458 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T01:54:39.458 INFO:tasks.workunit.client.0.trial112.stdout: "version": "20.2.0-636-g3b40640f", 2026-02-13T01:54:39.458 INFO:tasks.workunit.client.0.trial112.stdout: "release": "tentacle", 2026-02-13T01:54:39.458 INFO:tasks.workunit.client.0.trial112.stdout: "release_type": "stable" 2026-02-13T01:54:39.458 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T01:54:39.465 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:142: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2026-02-13T01:54:39.483 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T01:54:39.483 INFO:tasks.workunit.client.0.trial112.stdout: "name": "objxxx", 2026-02-13T01:54:39.483 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 5, 2026-02-13T01:54:39.483 INFO:tasks.workunit.client.0.trial112.stdout: "clones": [ 2026-02-13T01:54:39.483 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T01:54:39.483 INFO:tasks.workunit.client.0.trial112.stdout: "id": 2, 2026-02-13T01:54:39.484 INFO:tasks.workunit.client.0.trial112.stdout: "snapshots": [ 2026-02-13T01:54:39.484 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T01:54:39.484 INFO:tasks.workunit.client.0.trial112.stdout: "id": 1, 2026-02-13T01:54:39.484 INFO:tasks.workunit.client.0.trial112.stdout: "name": "snap01" 2026-02-13T01:54:39.484 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T01:54:39.484 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T01:54:39.484 INFO:tasks.workunit.client.0.trial112.stdout: "id": 2, 2026-02-13T01:54:39.484 INFO:tasks.workunit.client.0.trial112.stdout: "name": "snap02" 2026-02-13T01:54:39.484 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T01:54:39.484 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T01:54:39.484 INFO:tasks.workunit.client.0.trial112.stdout: "size": 5, 2026-02-13T01:54:39.485 INFO:tasks.workunit.client.0.trial112.stdout: "overlaps": [] 2026-02-13T01:54:39.485 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T01:54:39.485 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T01:54:39.485 INFO:tasks.workunit.client.0.trial112.stdout: "id": 3, 2026-02-13T01:54:39.485 INFO:tasks.workunit.client.0.trial112.stdout: "snapshots": [ 2026-02-13T01:54:39.485 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T01:54:39.485 INFO:tasks.workunit.client.0.trial112.stdout: "id": 3, 2026-02-13T01:54:39.485 INFO:tasks.workunit.client.0.trial112.stdout: "name": "snap13" 2026-02-13T01:54:39.485 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T01:54:39.485 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T01:54:39.485 INFO:tasks.workunit.client.0.trial112.stdout: "size": 6, 2026-02-13T01:54:39.485 INFO:tasks.workunit.client.0.trial112.stdout: "overlaps": [] 2026-02-13T01:54:39.486 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T01:54:39.486 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T01:54:39.486 INFO:tasks.workunit.client.0.trial112.stdout: "id": 5, 2026-02-13T01:54:39.486 INFO:tasks.workunit.client.0.trial112.stdout: "snapshots": [ 2026-02-13T01:54:39.486 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T01:54:39.486 INFO:tasks.workunit.client.0.trial112.stdout: "id": 4, 2026-02-13T01:54:39.486 INFO:tasks.workunit.client.0.trial112.stdout: "name": "snap24" 2026-02-13T01:54:39.486 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T01:54:39.486 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T01:54:39.486 INFO:tasks.workunit.client.0.trial112.stdout: "id": 5, 2026-02-13T01:54:39.486 INFO:tasks.workunit.client.0.trial112.stdout: "name": "snap25" 2026-02-13T01:54:39.486 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T01:54:39.486 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T01:54:39.487 INFO:tasks.workunit.client.0.trial112.stdout: "size": 6, 2026-02-13T01:54:39.487 INFO:tasks.workunit.client.0.trial112.stdout: "overlaps": [] 2026-02-13T01:54:39.487 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T01:54:39.487 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T01:54:39.487 INFO:tasks.workunit.client.0.trial112.stdout: "id": "head", 2026-02-13T01:54:39.487 INFO:tasks.workunit.client.0.trial112.stdout: "snapshots": [], 2026-02-13T01:54:39.487 INFO:tasks.workunit.client.0.trial112.stdout: "size": 6 2026-02-13T01:54:39.487 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T01:54:39.487 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T01:54:39.487 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T01:54:39.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:145: TEST_truncated_sna_record: ceph osd unset nodeep-scrub 2026-02-13T01:54:39.699 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is unset 2026-02-13T01:54:39.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:146: TEST_truncated_sna_record: ceph osd unset noscrub 2026-02-13T01:54:39.904 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is unset 2026-02-13T01:54:39.921 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2026-02-13T01:54:39.921 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: jq -r '.up[0]' 2026-02-13T01:54:40.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: local cur_prim=1 2026-02-13T01:54:40.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:150: TEST_truncated_sna_record: ceph pg dump pgs 2026-02-13T01:54:40.228 INFO:tasks.workunit.client.0.trial112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-13T01:54:40.228 INFO:tasks.workunit.client.0.trial112.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2026-02-13T01:54:38.168569+0000 22'7 36:65 [1,2,0] 1 [1,2,0] 1 22'7 2026-02-13T01:54:20.588325+0000 22'7 2026-02-13T01:54:20.588325+0000 0 0 periodic scrub scheduled @ 2026-02-14T01:54:20.588325+0000 4 0 2026-02-13T01:54:40.228 INFO:tasks.workunit.client.0.trial112.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-13T01:54:38.168631+0000 0'0 36:40 [0,1,2] 0 [0,1,2] 0 0'0 2026-02-13T01:54:15.096694+0000 0'0 2026-02-13T01:54:15.096694+0000 0 0 periodic scrub scheduled @ 2026-02-14T01:54:15.096694+0000 0 0 2026-02-13T01:54:40.228 INFO:tasks.workunit.client.0.trial112.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-13T01:54:38.168673+0000 0'0 36:48 [2,0,1] 2 [2,0,1] 2 0'0 2026-02-13T01:54:15.096694+0000 0'0 2026-02-13T01:54:15.096694+0000 0 0 periodic scrub scheduled @ 2026-02-14T01:54:15.096694+0000 0 0 2026-02-13T01:54:40.228 INFO:tasks.workunit.client.0.trial112.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-13T01:54:38.168701+0000 0'0 36:34 [1,0,2] 1 [1,0,2] 1 0'0 2026-02-13T01:54:15.096694+0000 0'0 2026-02-13T01:54:15.096694+0000 0 0 periodic scrub scheduled @ 2026-02-14T01:54:15.096694+0000 0 0 2026-02-13T01:54:40.228 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:40.228 INFO:tasks.workunit.client.0.trial112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-13T01:54:40.229 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T01:54:40.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:151: TEST_truncated_sna_record: sleep 2 2026-02-13T01:54:42.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:152: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2026-02-13T01:54:42.303 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T01:54:42.303 INFO:tasks.workunit.client.0.trial112.stdout: "deep": true, 2026-02-13T01:54:42.303 INFO:tasks.workunit.client.0.trial112.stdout: "must": true, 2026-02-13T01:54:42.303 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "0.000000" 2026-02-13T01:54:42.304 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T01:54:42.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:153: TEST_truncated_sna_record: sleep 5 2026-02-13T01:54:47.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:154: TEST_truncated_sna_record: ceph pg dump pgs 2026-02-13T01:54:47.463 INFO:tasks.workunit.client.0.trial112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-13T01:54:47.463 INFO:tasks.workunit.client.0.trial112.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2026-02-13T01:54:43.159401+0000 22'7 38:79 [1,2,0] 1 [1,2,0] 1 22'7 2026-02-13T01:54:43.159350+0000 22'7 2026-02-13T01:54:43.159350+0000 0 0 periodic scrub scheduled @ 2026-02-14T01:54:43.159350+0000 4 0 2026-02-13T01:54:47.463 INFO:tasks.workunit.client.0.trial112.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-13T01:54:38.168631+0000 0'0 38:45 [0,1,2] 0 [0,1,2] 0 0'0 2026-02-13T01:54:15.096694+0000 0'0 2026-02-13T01:54:15.096694+0000 0 0 periodic scrub scheduled @ 2026-02-14T01:54:15.096694+0000 0 0 2026-02-13T01:54:47.463 INFO:tasks.workunit.client.0.trial112.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-13T01:54:38.168673+0000 0'0 38:52 [2,0,1] 2 [2,0,1] 2 0'0 2026-02-13T01:54:15.096694+0000 0'0 2026-02-13T01:54:15.096694+0000 0 0 periodic scrub scheduled @ 2026-02-14T01:54:15.096694+0000 0 0 2026-02-13T01:54:47.463 INFO:tasks.workunit.client.0.trial112.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-13T01:54:38.168701+0000 0'0 38:40 [1,0,2] 1 [1,0,2] 1 0'0 2026-02-13T01:54:15.096694+0000 0'0 2026-02-13T01:54:15.096694+0000 0 0 periodic scrub scheduled @ 2026-02-14T01:54:15.096694+0000 0 0 2026-02-13T01:54:47.464 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:47.464 INFO:tasks.workunit.client.0.trial112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-13T01:54:47.464 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T01:54:47.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2026-02-13T01:54:47.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2026-02-13T01:54:47.473 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:54:43.155+0000 7f0820919640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2026-02-13T01:54:47.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:156: TEST_truncated_sna_record: grep -a -q ERR td/osd-mapper/osd.1.log 2026-02-13T01:54:47.475 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2026-02-13T01:54:47.475 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: wc -l 2026-02-13T01:54:47.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: local prev_err_cnt=1 2026-02-13T01:54:47.478 INFO:tasks.workunit.client.0.trial112.stdout:prev count: 1 2026-02-13T01:54:47.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:161: TEST_truncated_sna_record: echo 'prev count: 1' 2026-02-13T01:54:47.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:164: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2026-02-13T01:54:47.546 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T01:54:47.546 INFO:tasks.workunit.client.0.trial112.stdout: "deep": true, 2026-02-13T01:54:47.546 INFO:tasks.workunit.client.0.trial112.stdout: "must": true, 2026-02-13T01:54:47.546 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "0.000000" 2026-02-13T01:54:47.546 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T01:54:47.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:165: TEST_truncated_sna_record: sleep 5 2026-02-13T01:54:52.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:166: TEST_truncated_sna_record: ceph pg dump pgs 2026-02-13T01:54:52.701 INFO:tasks.workunit.client.0.trial112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-13T01:54:52.701 INFO:tasks.workunit.client.0.trial112.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2026-02-13T01:54:48.070511+0000 22'7 38:88 [1,2,0] 1 [1,2,0] 1 22'7 2026-02-13T01:54:48.070458+0000 22'7 2026-02-13T01:54:48.070458+0000 0 0 periodic scrub scheduled @ 2026-02-14T01:54:48.070458+0000 4 0 2026-02-13T01:54:52.701 INFO:tasks.workunit.client.0.trial112.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-13T01:54:38.168631+0000 0'0 38:47 [0,1,2] 0 [0,1,2] 0 0'0 2026-02-13T01:54:15.096694+0000 0'0 2026-02-13T01:54:15.096694+0000 0 0 periodic scrub scheduled @ 2026-02-14T01:54:15.096694+0000 0 0 2026-02-13T01:54:52.701 INFO:tasks.workunit.client.0.trial112.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-13T01:54:38.168673+0000 0'0 38:54 [2,0,1] 2 [2,0,1] 2 0'0 2026-02-13T01:54:15.096694+0000 0'0 2026-02-13T01:54:15.096694+0000 0 0 periodic scrub scheduled @ 2026-02-14T01:54:15.096694+0000 0 0 2026-02-13T01:54:52.701 INFO:tasks.workunit.client.0.trial112.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-13T01:54:38.168701+0000 0'0 38:42 [1,0,2] 1 [1,0,2] 1 0'0 2026-02-13T01:54:15.096694+0000 0'0 2026-02-13T01:54:15.096694+0000 0 0 periodic scrub scheduled @ 2026-02-14T01:54:15.096694+0000 0 0 2026-02-13T01:54:52.701 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:54:52.702 INFO:tasks.workunit.client.0.trial112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-13T01:54:52.702 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T01:54:52.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2026-02-13T01:54:52.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2026-02-13T01:54:52.711 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:54:43.155+0000 7f0820919640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2026-02-13T01:54:52.712 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2026-02-13T01:54:52.712 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: wc -l 2026-02-13T01:54:52.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: local current_err_cnt=1 2026-02-13T01:54:52.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2026-02-13T01:54:52.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: echo 'current count: 1' 2026-02-13T01:54:52.715 INFO:tasks.workunit.client.0.trial112.stdout:current count: 1 2026-02-13T01:54:52.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:170: TEST_truncated_sna_record: (( current_err_cnt == prev_err_cnt )) 2026-02-13T01:54:52.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:171: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2026-02-13T01:54:52.716 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T01:54:52.716 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T01:54:52.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T01:54:52.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T01:54:52.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T01:54:53.024 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T01:54:53.024 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:172: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2026-02-13T01:54:53.024 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2026-02-13T01:54:53.024 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2026-02-13T01:54:53.024 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_' 2026-02-13T01:54:53.024 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:174: TEST_truncated_sna_record: awk -e '{print $2;}' 2026-02-13T01:54:53.243 INFO:tasks.workunit.client.0.trial112.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2026-02-13T01:54:53.243 INFO:tasks.workunit.client.0.trial112.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2026-02-13T01:54:53.244 INFO:tasks.workunit.client.0.trial112.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2026-02-13T01:54:53.244 INFO:tasks.workunit.client.0.trial112.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-13T01:54:53.244 INFO:tasks.workunit.client.0.trial112.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2026-02-13T01:54:53.244 INFO:tasks.workunit.client.0.trial112.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2026-02-13T01:54:53.245 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2026-02-13T01:54:53.245 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2026-02-13T01:54:53.245 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: awk -e '{print $2;}' 2026-02-13T01:54:53.245 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: wc -l 2026-02-13T01:54:53.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: local num_sna_full=5 2026-02-13T01:54:53.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:177: TEST_truncated_sna_record: (( num_sna_full == num_sna_b4 )) 2026-02-13T01:54:53.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:178: TEST_truncated_sna_record: return 0 2026-02-13T01:54:53.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:24: run: teardown td/osd-mapper 2026-02-13T01:54:53.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-mapper 2026-02-13T01:54:53.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T01:54:53.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-mapper KILL 2026-02-13T01:54:53.461 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T01:54:53.461 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T01:54:53.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T01:54:53.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T01:54:53.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T01:54:53.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T01:54:53.567 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T01:54:53.568 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T01:54:53.568 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T01:54:53.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T01:54:53.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T01:54:53.570 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T01:54:53.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T01:54:53.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T01:54:53.572 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T01:54:53.572 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T01:54:53.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T01:54:53.574 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T01:54:53.574 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-mapper 2026-02-13T01:54:53.587 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T01:54:53.587 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:54:53.587 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7480 2026-02-13T01:54:53.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.7480 2026-02-13T01:54:53.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T01:54:53.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T01:54:53.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: main: code=0 2026-02-13T01:54:53.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: teardown td/osd-mapper 0 2026-02-13T01:54:53.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-mapper 2026-02-13T01:54:53.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-13T01:54:53.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-mapper KILL 2026-02-13T01:54:53.590 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T01:54:53.590 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T01:54:53.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T01:54:53.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T01:54:53.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T01:54:53.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T01:54:53.592 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T01:54:53.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T01:54:53.592 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T01:54:53.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T01:54:53.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T01:54:53.594 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T01:54:53.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T01:54:53.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T01:54:53.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T01:54:53.596 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T01:54:53.597 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T01:54:53.598 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-13T01:54:53.598 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-mapper 2026-02-13T01:54:53.599 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T01:54:53.599 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:54:53.599 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7480 2026-02-13T01:54:53.599 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.7480 2026-02-13T01:54:53.600 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T01:54:53.600 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T01:54:53.600 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: main: return 0 2026-02-13T01:54:53.602 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-13T01:54:53.602 DEBUG:teuthology.orchestra.run.trial112:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-13T01:54:53.651 INFO:tasks.workunit:Running workunit scrub/osd-recovery-scrub.sh... 2026-02-13T01:54:53.651 DEBUG:teuthology.orchestra.run.trial112:workunit test scrub/osd-recovery-scrub.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh 2026-02-13T01:54:53.704 INFO:tasks.workunit.client.0.trial112.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-13T01:54:53.707 INFO:tasks.workunit.client.0.trial112.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-13T01:54:53.707 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-13T01:54:53.707 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-13T01:54:53.707 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export PYTHONWARNINGS=ignore 2026-02-13T01:54:53.707 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: PYTHONWARNINGS=ignore 2026-02-13T01:54:53.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: export CEPH_CONF=/dev/null 2026-02-13T01:54:53.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: CEPH_CONF=/dev/null 2026-02-13T01:54:53.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: main: unset CEPH_ARGS 2026-02-13T01:54:53.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: local code 2026-02-13T01:54:53.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: run td/osd-recovery-scrub 2026-02-13T01:54:53.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:20: run: local dir=td/osd-recovery-scrub 2026-02-13T01:54:53.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:21: run: shift 2026-02-13T01:54:53.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: export CEPH_MON=127.0.0.1:7124 2026-02-13T01:54:53.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: CEPH_MON=127.0.0.1:7124 2026-02-13T01:54:53.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:24: run: export CEPH_ARGS 2026-02-13T01:54:53.708 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: uuidgen 2026-02-13T01:54:53.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: CEPH_ARGS+='--fsid=66065daa-84cf-46ff-ac1f-20f488566d93 --auth-supported=none ' 2026-02-13T01:54:53.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2026-02-13T01:54:53.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:27: run: CEPH_ARGS+='--osd-op-queue=wpq ' 2026-02-13T01:54:53.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:29: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2026-02-13T01:54:53.710 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: set 2026-02-13T01:54:53.710 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-13T01:54:53.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: local funcs=TEST_recovery_scrub_1 2026-02-13T01:54:53.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2026-02-13T01:54:53.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_1 td/osd-recovery-scrub 2026-02-13T01:54:53.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:39: TEST_recovery_scrub_1: local dir=td/osd-recovery-scrub 2026-02-13T01:54:53.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:40: TEST_recovery_scrub_1: local poolname=test 2026-02-13T01:54:53.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:42: TEST_recovery_scrub_1: TESTDATA=testdata.25784 2026-02-13T01:54:53.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:43: TEST_recovery_scrub_1: OSDS=4 2026-02-13T01:54:53.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:44: TEST_recovery_scrub_1: PGS=1 2026-02-13T01:54:53.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:45: TEST_recovery_scrub_1: OBJECTS=100 2026-02-13T01:54:53.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:46: TEST_recovery_scrub_1: ERRORS=0 2026-02-13T01:54:53.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:48: TEST_recovery_scrub_1: setup td/osd-recovery-scrub 2026-02-13T01:54:53.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-recovery-scrub 2026-02-13T01:54:53.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-recovery-scrub 2026-02-13T01:54:53.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-recovery-scrub 2026-02-13T01:54:53.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T01:54:53.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-recovery-scrub KILL 2026-02-13T01:54:53.713 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T01:54:53.713 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T01:54:53.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T01:54:53.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T01:54:53.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T01:54:53.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T01:54:53.714 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T01:54:53.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T01:54:53.715 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T01:54:53.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T01:54:53.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T01:54:53.716 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T01:54:53.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T01:54:53.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T01:54:53.718 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T01:54:53.718 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T01:54:53.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T01:54:53.720 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T01:54:53.720 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-recovery-scrub 2026-02-13T01:54:53.722 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T01:54:53.722 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:54:53.722 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.25784 2026-02-13T01:54:53.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.25784 2026-02-13T01:54:53.723 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T01:54:53.723 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T01:54:53.723 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-recovery-scrub 2026-02-13T01:54:53.725 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T01:54:53.725 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:54:53.725 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.25784 2026-02-13T01:54:53.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.25784 2026-02-13T01:54:53.726 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T01:54:53.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-13T01:54:53.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-13T01:54:53.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T01:54:53.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2026-02-13T01:54:53.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:49: TEST_recovery_scrub_1: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-13T01:54:53.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-recovery-scrub 2026-02-13T01:54:53.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-13T01:54:53.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-13T01:54:53.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-13T01:54:53.729 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-recovery-scrub/a 2026-02-13T01:54:53.729 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-13T01:54:53.753 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-13T01:54:53.753 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:54:53.753 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:54:53.753 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:54:53.753 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:54:53.753 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.25784 2026-02-13T01:54:53.753 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.25784/$cluster-$name.asok' 2026-02-13T01:54:53.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.25784/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-13T01:54:53.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-13T01:54:53.784 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-13T01:54:53.784 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T01:54:53.784 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T01:54:53.784 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-13T01:54:53.784 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-13T01:54:53.784 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T01:54:53.784 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T01:54:53.784 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T01:54:53.785 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T01:54:53.785 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:54:53.785 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.25784 2026-02-13T01:54:53.785 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.25784/ceph-mon.a.asok 2026-02-13T01:54:53.786 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T01:54:53.786 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.25784/ceph-mon.a.asok config get fsid 2026-02-13T01:54:53.841 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-13T01:54:53.841 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T01:54:53.841 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T01:54:53.841 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-13T01:54:53.841 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-13T01:54:53.842 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T01:54:53.843 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T01:54:53.843 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T01:54:53.843 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T01:54:53.843 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:54:53.843 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.25784 2026-02-13T01:54:53.843 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.25784/ceph-mon.a.asok 2026-02-13T01:54:53.843 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T01:54:53.843 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.25784/ceph-mon.a.asok config get mon_host 2026-02-13T01:54:53.898 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:50: TEST_recovery_scrub_1: run_mgr td/osd-recovery-scrub x 2026-02-13T01:54:53.898 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-recovery-scrub 2026-02-13T01:54:53.898 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-13T01:54:53.898 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-13T01:54:53.899 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-13T01:54:53.899 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-recovery-scrub/x 2026-02-13T01:54:53.899 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-13T01:54:53.993 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-13T01:54:53.993 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:54:53.993 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:54:53.993 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:54:53.993 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:54:53.993 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.25784 2026-02-13T01:54:53.993 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.25784/$cluster-$name.asok' 2026-02-13T01:54:53.994 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T01:54:53.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.25784/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T01:54:54.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 ' 2026-02-13T01:54:54.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:52: TEST_recovery_scrub_1: ceph_osd_args+='--osd_scrub_backoff_ratio=0 ' 2026-02-13T01:54:54.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:53: TEST_recovery_scrub_1: ceph_osd_args+='--osd_stats_update_period_not_scrubbing=3 ' 2026-02-13T01:54:54.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:54: TEST_recovery_scrub_1: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2026-02-13T01:54:54.013 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: expr 4 - 1 2026-02-13T01:54:54.014 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: seq 0 3 2026-02-13T01:54:54.017 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T01:54:54.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=false 2026-02-13T01:54:54.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-recovery-scrub 2026-02-13T01:54:54.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T01:54:54.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-13T01:54:54.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T01:54:54.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-recovery-scrub/0 2026-02-13T01:54:54.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=66065daa-84cf-46ff-ac1f-20f488566d93 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2026-02-13T01:54:54.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T01:54:54.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T01:54:54.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T01:54:54.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2026-02-13T01:54:54.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2026-02-13T01:54:54.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T01:54:54.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T01:54:54.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2026-02-13T01:54:54.019 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T01:54:54.019 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:54:54.019 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:54:54.019 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:54:54.019 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:54:54.019 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.25784 2026-02-13T01:54:54.019 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.25784/$cluster-$name.asok' 2026-02-13T01:54:54.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25784/$cluster-$name.asok' 2026-02-13T01:54:54.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T01:54:54.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T01:54:54.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T01:54:54.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2026-02-13T01:54:54.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2026-02-13T01:54:54.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T01:54:54.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T01:54:54.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T01:54:54.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T01:54:54.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T01:54:54.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2026-02-13T01:54:54.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-recovery-scrub/0 2026-02-13T01:54:54.020 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T01:54:54.021 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a902eb5c-69a1-4142-8c89-4e51c90a5106 2026-02-13T01:54:54.021 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 a902eb5c-69a1-4142-8c89-4e51c90a5106' 2026-02-13T01:54:54.021 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 a902eb5c-69a1-4142-8c89-4e51c90a5106 2026-02-13T01:54:54.022 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T01:54:54.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBuhI5paEqOARAAu4LnjQTqv4Ufgwk6bND/qQ== 2026-02-13T01:54:54.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBuhI5paEqOARAAu4LnjQTqv4Ufgwk6bND/qQ=="}' 2026-02-13T01:54:54.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a902eb5c-69a1-4142-8c89-4e51c90a5106 -i td/osd-recovery-scrub/0/new.json 2026-02-13T01:54:54.121 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T01:54:54.129 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-recovery-scrub/0/new.json 2026-02-13T01:54:54.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=66065daa-84cf-46ff-ac1f-20f488566d93 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.25784/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQBuhI5paEqOARAAu4LnjQTqv4Ufgwk6bND/qQ== --osd-uuid a902eb5c-69a1-4142-8c89-4e51c90a5106 2026-02-13T01:54:54.145 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:54.135+0000 7f32d6d91a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:54:54.147 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:54.139+0000 7f32d6d91a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:54:54.147 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:54.139+0000 7f32d6d91a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:54:54.148 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:54.139+0000 7f32d6d91a40 -1 bdev(0x55cab5088c00 td/osd-recovery-scrub/0/block) open stat got: (1) Operation not permitted 2026-02-13T01:54:54.148 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:54.139+0000 7f32d6d91a40 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2026-02-13T01:54:54.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2026-02-13T01:54:54.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T01:54:54.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-13T01:54:54.844 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T01:54:54.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T01:54:54.941 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T01:54:54.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-13T01:54:54.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=66065daa-84cf-46ff-ac1f-20f488566d93 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.25784/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2026-02-13T01:54:54.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T01:54:54.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T01:54:54.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T01:54:54.956 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:54.947+0000 7fe3d123ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:54:54.957 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:54.947+0000 7fe3d123ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:54:54.958 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:54.947+0000 7fe3d123ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:54:55.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-13T01:54:55.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T01:54:55.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T01:54:55.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T01:54:55.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T01:54:55.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:54:55.039 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T01:54:55.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T01:54:55.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:54:55.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T01:54:55.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:54:55.368 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:55.359+0000 7fe3d123ba40 -1 Falling back to public interface 2026-02-13T01:54:56.052 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:56.043+0000 7fe3d123ba40 -1 osd.0 0 log_to_monitors true 2026-02-13T01:54:56.132 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:54:56.132 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T01:54:56.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:54:56.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T01:54:56.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:54:56.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T01:54:56.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:54:57.253 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:54:57.253 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:54:57.254 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T01:54:57.254 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T01:54:57.254 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:54:57.254 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T01:54:57.414 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/999910389,v1:127.0.0.1:6801/999910389] [v2:127.0.0.1:6802/999910389,v1:127.0.0.1:6803/999910389] exists,up a902eb5c-69a1-4142-8c89-4e51c90a5106 2026-02-13T01:54:57.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T01:54:57.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T01:54:57.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T01:54:57.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T01:54:57.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=false 2026-02-13T01:54:57.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-recovery-scrub 2026-02-13T01:54:57.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T01:54:57.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-13T01:54:57.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T01:54:57.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-recovery-scrub/1 2026-02-13T01:54:57.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=66065daa-84cf-46ff-ac1f-20f488566d93 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2026-02-13T01:54:57.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T01:54:57.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T01:54:57.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T01:54:57.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2026-02-13T01:54:57.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2026-02-13T01:54:57.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T01:54:57.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T01:54:57.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2026-02-13T01:54:57.419 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T01:54:57.419 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:54:57.419 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:54:57.419 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:54:57.419 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:54:57.419 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.25784 2026-02-13T01:54:57.419 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.25784/$cluster-$name.asok' 2026-02-13T01:54:57.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25784/$cluster-$name.asok' 2026-02-13T01:54:57.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T01:54:57.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T01:54:57.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T01:54:57.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2026-02-13T01:54:57.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2026-02-13T01:54:57.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T01:54:57.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T01:54:57.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T01:54:57.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T01:54:57.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T01:54:57.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2026-02-13T01:54:57.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-recovery-scrub/1 2026-02-13T01:54:57.421 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T01:54:57.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4c369dd5-fb45-4f7e-b1be-ae481d99f6db 2026-02-13T01:54:57.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 4c369dd5-fb45-4f7e-b1be-ae481d99f6db' 2026-02-13T01:54:57.422 INFO:tasks.workunit.client.0.trial112.stdout:add osd1 4c369dd5-fb45-4f7e-b1be-ae481d99f6db 2026-02-13T01:54:57.422 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T01:54:57.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBxhI5pI4RRGRAAQ/piLqunYQz/oZ8qklGDDw== 2026-02-13T01:54:57.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBxhI5pI4RRGRAAQ/piLqunYQz/oZ8qklGDDw=="}' 2026-02-13T01:54:57.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4c369dd5-fb45-4f7e-b1be-ae481d99f6db -i td/osd-recovery-scrub/1/new.json 2026-02-13T01:54:57.595 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T01:54:57.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-recovery-scrub/1/new.json 2026-02-13T01:54:57.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=66065daa-84cf-46ff-ac1f-20f488566d93 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.25784/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQBxhI5pI4RRGRAAQ/piLqunYQz/oZ8qklGDDw== --osd-uuid 4c369dd5-fb45-4f7e-b1be-ae481d99f6db 2026-02-13T01:54:57.621 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:57.611+0000 7f995e0c2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:54:57.623 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:57.615+0000 7f995e0c2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:54:57.624 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:57.615+0000 7f995e0c2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:54:57.625 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:57.615+0000 7f995e0c2a40 -1 bdev(0x561fac238c00 td/osd-recovery-scrub/1/block) open stat got: (1) Operation not permitted 2026-02-13T01:54:57.625 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:57.615+0000 7f995e0c2a40 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2026-02-13T01:54:58.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2026-02-13T01:54:58.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T01:54:58.262 INFO:tasks.workunit.client.0.trial112.stdout:adding osd1 key to auth repository 2026-02-13T01:54:58.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-13T01:54:58.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T01:54:58.475 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T01:54:58.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-13T01:54:58.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=66065daa-84cf-46ff-ac1f-20f488566d93 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.25784/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2026-02-13T01:54:58.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T01:54:58.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T01:54:58.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T01:54:58.499 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:58.491+0000 7fe5ae607a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:54:58.501 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:58.491+0000 7fe5ae607a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:54:58.502 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:58.491+0000 7fe5ae607a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:54:58.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-13T01:54:58.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T01:54:58.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T01:54:58.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T01:54:58.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T01:54:58.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:54:58.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T01:54:58.636 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T01:54:58.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:54:58.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T01:54:58.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:54:58.932 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:58.923+0000 7fe5ae607a40 -1 Falling back to public interface 2026-02-13T01:54:59.548 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:54:59.539+0000 7fe5ae607a40 -1 osd.1 0 log_to_monitors true 2026-02-13T01:54:59.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:54:59.795 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:54:59.795 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T01:54:59.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T01:54:59.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:54:59.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T01:54:59.957 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2974687472,v1:127.0.0.1:6811/2974687472] [v2:127.0.0.1:6812/2974687472,v1:127.0.0.1:6813/2974687472] exists,up 4c369dd5-fb45-4f7e-b1be-ae481d99f6db 2026-02-13T01:54:59.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T01:54:59.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T01:54:59.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T01:54:59.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T01:54:59.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=false 2026-02-13T01:54:59.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-recovery-scrub 2026-02-13T01:54:59.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T01:54:59.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-13T01:54:59.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T01:54:59.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-recovery-scrub/2 2026-02-13T01:54:59.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=66065daa-84cf-46ff-ac1f-20f488566d93 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2026-02-13T01:54:59.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T01:54:59.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T01:54:59.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T01:54:59.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2026-02-13T01:54:59.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2026-02-13T01:54:59.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T01:54:59.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T01:54:59.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2026-02-13T01:54:59.962 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T01:54:59.962 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:54:59.962 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:54:59.962 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:54:59.962 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:54:59.962 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.25784 2026-02-13T01:54:59.962 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.25784/$cluster-$name.asok' 2026-02-13T01:54:59.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25784/$cluster-$name.asok' 2026-02-13T01:54:59.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T01:54:59.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T01:54:59.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T01:54:59.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2026-02-13T01:54:59.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2026-02-13T01:54:59.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T01:54:59.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T01:54:59.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T01:54:59.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T01:54:59.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T01:54:59.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2026-02-13T01:54:59.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-recovery-scrub/2 2026-02-13T01:54:59.964 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T01:54:59.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d8e9e69a-750f-4ad6-9804-bba1ae574c7d 2026-02-13T01:54:59.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 d8e9e69a-750f-4ad6-9804-bba1ae574c7d' 2026-02-13T01:54:59.964 INFO:tasks.workunit.client.0.trial112.stdout:add osd2 d8e9e69a-750f-4ad6-9804-bba1ae574c7d 2026-02-13T01:54:59.965 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T01:54:59.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBzhI5pAgi/ORAAoMJZ6ChPgnlhV5te6gmGZw== 2026-02-13T01:54:59.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBzhI5pAgi/ORAAoMJZ6ChPgnlhV5te6gmGZw=="}' 2026-02-13T01:54:59.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d8e9e69a-750f-4ad6-9804-bba1ae574c7d -i td/osd-recovery-scrub/2/new.json 2026-02-13T01:55:00.143 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T01:55:00.152 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-recovery-scrub/2/new.json 2026-02-13T01:55:00.152 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=66065daa-84cf-46ff-ac1f-20f488566d93 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.25784/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQBzhI5pAgi/ORAAoMJZ6ChPgnlhV5te6gmGZw== --osd-uuid d8e9e69a-750f-4ad6-9804-bba1ae574c7d 2026-02-13T01:55:00.169 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:00.159+0000 7fe513f11a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:55:00.170 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:00.159+0000 7fe513f11a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:55:00.171 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:00.163+0000 7fe513f11a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:55:00.171 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:00.163+0000 7fe513f11a40 -1 bdev(0x5622e5ba4c00 td/osd-recovery-scrub/2/block) open stat got: (1) Operation not permitted 2026-02-13T01:55:00.171 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:00.163+0000 7fe513f11a40 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2026-02-13T01:55:00.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2026-02-13T01:55:00.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T01:55:00.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-13T01:55:00.863 INFO:tasks.workunit.client.0.trial112.stdout:adding osd2 key to auth repository 2026-02-13T01:55:00.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T01:55:01.069 INFO:tasks.workunit.client.0.trial112.stdout:start osd.2 2026-02-13T01:55:01.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-13T01:55:01.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=66065daa-84cf-46ff-ac1f-20f488566d93 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.25784/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2026-02-13T01:55:01.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T01:55:01.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T01:55:01.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T01:55:01.094 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:01.083+0000 7f164c1f8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:55:01.095 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:01.087+0000 7f164c1f8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:55:01.096 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:01.087+0000 7f164c1f8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:55:01.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-13T01:55:01.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T01:55:01.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T01:55:01.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T01:55:01.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T01:55:01.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:55:01.227 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T01:55:01.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T01:55:01.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:55:01.228 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T01:55:01.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:55:01.404 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:01.395+0000 7f164c1f8a40 -1 Falling back to public interface 2026-02-13T01:55:02.029 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:02.019+0000 7f164c1f8a40 -1 osd.2 0 log_to_monitors true 2026-02-13T01:55:02.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:55:02.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:55:02.392 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T01:55:02.393 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T01:55:02.393 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:55:02.393 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T01:55:02.556 INFO:tasks.workunit.client.0.trial112.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3259475373,v1:127.0.0.1:6819/3259475373] [v2:127.0.0.1:6820/3259475373,v1:127.0.0.1:6821/3259475373] exists,up d8e9e69a-750f-4ad6-9804-bba1ae574c7d 2026-02-13T01:55:02.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T01:55:02.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T01:55:02.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T01:55:02.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T01:55:02.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=false 2026-02-13T01:55:02.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-recovery-scrub 2026-02-13T01:55:02.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T01:55:02.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-13T01:55:02.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T01:55:02.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-recovery-scrub/3 2026-02-13T01:55:02.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=66065daa-84cf-46ff-ac1f-20f488566d93 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2026-02-13T01:55:02.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T01:55:02.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T01:55:02.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T01:55:02.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2026-02-13T01:55:02.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2026-02-13T01:55:02.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T01:55:02.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T01:55:02.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2026-02-13T01:55:02.561 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T01:55:02.561 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:55:02.561 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:55:02.561 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:55:02.561 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:55:02.561 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.25784 2026-02-13T01:55:02.561 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.25784/$cluster-$name.asok' 2026-02-13T01:55:02.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25784/$cluster-$name.asok' 2026-02-13T01:55:02.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T01:55:02.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T01:55:02.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T01:55:02.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2026-02-13T01:55:02.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2026-02-13T01:55:02.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T01:55:02.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T01:55:02.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T01:55:02.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T01:55:02.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T01:55:02.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2026-02-13T01:55:02.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-recovery-scrub/3 2026-02-13T01:55:02.563 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T01:55:02.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ba09d6a2-1797-44d9-b097-945e083c0607 2026-02-13T01:55:02.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 ba09d6a2-1797-44d9-b097-945e083c0607' 2026-02-13T01:55:02.563 INFO:tasks.workunit.client.0.trial112.stdout:add osd3 ba09d6a2-1797-44d9-b097-945e083c0607 2026-02-13T01:55:02.564 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T01:55:02.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB2hI5pk6HdIRAAoxqb+4w+HGWxKGlA9yeQqQ== 2026-02-13T01:55:02.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB2hI5pk6HdIRAAoxqb+4w+HGWxKGlA9yeQqQ=="}' 2026-02-13T01:55:02.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ba09d6a2-1797-44d9-b097-945e083c0607 -i td/osd-recovery-scrub/3/new.json 2026-02-13T01:55:02.742 INFO:tasks.workunit.client.0.trial112.stdout:3 2026-02-13T01:55:02.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-recovery-scrub/3/new.json 2026-02-13T01:55:02.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=66065daa-84cf-46ff-ac1f-20f488566d93 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.25784/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQB2hI5pk6HdIRAAoxqb+4w+HGWxKGlA9yeQqQ== --osd-uuid ba09d6a2-1797-44d9-b097-945e083c0607 2026-02-13T01:55:02.769 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:02.759+0000 7f76cd6d6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:55:02.770 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:02.759+0000 7f76cd6d6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:55:02.771 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:02.763+0000 7f76cd6d6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:55:02.772 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:02.763+0000 7f76cd6d6a40 -1 bdev(0x55edba798c00 td/osd-recovery-scrub/3/block) open stat got: (1) Operation not permitted 2026-02-13T01:55:02.772 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:02.763+0000 7f76cd6d6a40 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2026-02-13T01:55:03.465 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2026-02-13T01:55:03.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T01:55:03.466 INFO:tasks.workunit.client.0.trial112.stdout:adding osd3 key to auth repository 2026-02-13T01:55:03.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-13T01:55:03.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T01:55:03.672 INFO:tasks.workunit.client.0.trial112.stdout:start osd.3 2026-02-13T01:55:03.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-13T01:55:03.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=66065daa-84cf-46ff-ac1f-20f488566d93 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.25784/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2026-02-13T01:55:03.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T01:55:03.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T01:55:03.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T01:55:03.696 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:03.687+0000 7f0af1ca2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:55:03.698 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:03.687+0000 7f0af1ca2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:55:03.699 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:03.691+0000 7f0af1ca2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:55:03.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-13T01:55:03.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T01:55:03.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-13T01:55:03.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T01:55:03.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T01:55:03.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:55:03.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T01:55:03.832 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T01:55:03.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:55:03.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T01:55:03.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:55:04.044 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:04.035+0000 7f0af1ca2a40 -1 Falling back to public interface 2026-02-13T01:55:04.744 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:04.735+0000 7f0af1ca2a40 -1 osd.3 0 log_to_monitors true 2026-02-13T01:55:04.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:55:04.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:55:04.993 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T01:55:04.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T01:55:04.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:55:04.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T01:55:05.152 INFO:tasks.workunit.client.0.trial112.stdout:osd.3 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3412609621,v1:127.0.0.1:6827/3412609621] [v2:127.0.0.1:6828/3412609621,v1:127.0.0.1:6829/3412609621] exists,up ba09d6a2-1797-44d9-b097-945e083c0607 2026-02-13T01:55:05.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T01:55:05.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T01:55:05.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T01:55:05.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:61: TEST_recovery_scrub_1: create_pool test 1 1 2026-02-13T01:55:05.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2026-02-13T01:55:05.359 INFO:tasks.workunit.client.0.trial112.stderr:pool 'test' created 2026-02-13T01:55:05.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T01:55:06.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:62: TEST_recovery_scrub_1: wait_for_clean 2026-02-13T01:55:06.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T01:55:06.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T01:55:06.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T01:55:06.372 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T01:55:06.372 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T01:55:06.373 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T01:55:06.373 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T01:55:06.373 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T01:55:06.373 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T01:55:06.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T01:55:06.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T01:55:06.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T01:55:06.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T01:55:06.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T01:55:06.473 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T01:55:06.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T01:55:06.641 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:55:06.641 INFO:tasks.workunit.client.0.trial112.stderr:2 2026-02-13T01:55:06.641 INFO:tasks.workunit.client.0.trial112.stderr:3' 2026-02-13T01:55:06.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T01:55:06.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:55:06.642 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T01:55:06.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836484 2026-02-13T01:55:06.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836484 2026-02-13T01:55:06.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484' 2026-02-13T01:55:06.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:55:06.718 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T01:55:06.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705667 2026-02-13T01:55:06.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705667 2026-02-13T01:55:06.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-38654705667' 2026-02-13T01:55:06.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:55:06.793 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T01:55:06.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574850 2026-02-13T01:55:06.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574850 2026-02-13T01:55:06.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-38654705667 2-55834574850' 2026-02-13T01:55:06.866 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:55:06.866 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T01:55:06.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=73014444034 2026-02-13T01:55:06.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 73014444034 2026-02-13T01:55:06.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-38654705667 2-55834574850 3-73014444034' 2026-02-13T01:55:06.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:55:06.939 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836484 2026-02-13T01:55:06.939 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:55:06.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T01:55:06.941 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836484 2026-02-13T01:55:06.941 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:55:06.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836484 2026-02-13T01:55:06.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-13T01:55:06.943 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836484 2026-02-13T01:55:06.943 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:55:07.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836484 2026-02-13T01:55:07.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:55:08.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T01:55:08.116 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:55:08.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836484 2026-02-13T01:55:08.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:55:09.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T01:55:09.277 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:55:09.442 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836484 2026-02-13T01:55:09.442 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:55:09.443 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705667 2026-02-13T01:55:09.443 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:55:09.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T01:55:09.445 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705667 2026-02-13T01:55:09.445 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:55:09.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705667 2026-02-13T01:55:09.446 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705667 2026-02-13T01:55:09.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705667' 2026-02-13T01:55:09.447 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T01:55:09.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705667 -lt 38654705667 2026-02-13T01:55:09.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:55:09.608 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574850 2026-02-13T01:55:09.609 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:55:09.610 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T01:55:09.610 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574850 2026-02-13T01:55:09.610 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:55:09.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574850 2026-02-13T01:55:09.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574850' 2026-02-13T01:55:09.612 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574850 2026-02-13T01:55:09.612 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T01:55:09.773 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574851 -lt 55834574850 2026-02-13T01:55:09.773 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:55:09.774 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-73014444034 2026-02-13T01:55:09.774 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:55:09.775 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T01:55:09.776 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-73014444034 2026-02-13T01:55:09.776 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:55:09.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=73014444034 2026-02-13T01:55:09.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 73014444034' 2026-02-13T01:55:09.778 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.3 seq 73014444034 2026-02-13T01:55:09.779 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T01:55:09.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 73014444034 -lt 73014444034 2026-02-13T01:55:09.935 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T01:55:09.936 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:55:09.936 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:55:10.125 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T01:55:10.125 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T01:55:10.126 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T01:55:10.126 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T01:55:10.126 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T01:55:10.126 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T01:55:10.126 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T01:55:10.126 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T01:55:10.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T01:55:10.283 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T01:55:10.283 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:55:10.284 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:55:10.474 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T01:55:10.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T01:55:10.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T01:55:10.476 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: ceph osd dump 2026-02-13T01:55:10.476 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: grep '^pool.*['\'']test['\'']' 2026-02-13T01:55:10.476 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: awk '{ print $2 }' 2026-02-13T01:55:10.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: poolid=1 2026-02-13T01:55:10.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:65: TEST_recovery_scrub_1: ceph pg dump pgs 2026-02-13T01:55:10.789 INFO:tasks.workunit.client.0.trial112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-13T01:55:10.789 INFO:tasks.workunit.client.0.trial112.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-13T01:55:05.687424+0000 0'0 20:12 [1] 1 [1] 1 0'0 2026-02-13T01:55:05.354036+0000 0'0 2026-02-13T01:55:05.354036+0000 0 0 periodic scrub scheduled @ 2026-02-14T07:25:44.972525+0000 0 0 2026-02-13T01:55:10.789 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:55:10.789 INFO:tasks.workunit.client.0.trial112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-13T01:55:10.789 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T01:55:10.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub_1: dd if=/dev/urandom of=testdata.25784 bs=1M count=50 2026-02-13T01:55:10.940 INFO:tasks.workunit.client.0.trial112.stderr:50+0 records in 2026-02-13T01:55:10.940 INFO:tasks.workunit.client.0.trial112.stderr:50+0 records out 2026-02-13T01:55:10.940 INFO:tasks.workunit.client.0.trial112.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.141465 s, 371 MB/s 2026-02-13T01:55:10.941 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: seq 1 100 2026-02-13T01:55:10.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:10.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj1 testdata.25784 2026-02-13T01:55:11.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:11.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj2 testdata.25784 2026-02-13T01:55:11.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:11.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj3 testdata.25784 2026-02-13T01:55:11.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:11.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj4 testdata.25784 2026-02-13T01:55:11.253 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:11.254 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj5 testdata.25784 2026-02-13T01:55:11.332 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:11.332 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj6 testdata.25784 2026-02-13T01:55:11.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:11.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj7 testdata.25784 2026-02-13T01:55:11.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:11.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj8 testdata.25784 2026-02-13T01:55:11.581 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:11.581 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj9 testdata.25784 2026-02-13T01:55:11.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:11.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj10 testdata.25784 2026-02-13T01:55:11.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:11.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj11 testdata.25784 2026-02-13T01:55:11.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:11.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj12 testdata.25784 2026-02-13T01:55:11.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:11.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj13 testdata.25784 2026-02-13T01:55:11.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:11.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj14 testdata.25784 2026-02-13T01:55:12.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:12.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj15 testdata.25784 2026-02-13T01:55:12.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:12.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj16 testdata.25784 2026-02-13T01:55:12.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:12.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj17 testdata.25784 2026-02-13T01:55:12.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:12.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj18 testdata.25784 2026-02-13T01:55:12.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:12.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj19 testdata.25784 2026-02-13T01:55:12.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:12.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj20 testdata.25784 2026-02-13T01:55:12.548 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:12.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj21 testdata.25784 2026-02-13T01:55:12.626 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:12.627 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj22 testdata.25784 2026-02-13T01:55:12.707 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:12.707 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj23 testdata.25784 2026-02-13T01:55:12.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:12.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj24 testdata.25784 2026-02-13T01:55:12.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:12.859 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj25 testdata.25784 2026-02-13T01:55:12.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:12.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj26 testdata.25784 2026-02-13T01:55:13.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:13.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj27 testdata.25784 2026-02-13T01:55:13.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:13.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj28 testdata.25784 2026-02-13T01:55:13.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:13.170 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj29 testdata.25784 2026-02-13T01:55:13.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:13.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj30 testdata.25784 2026-02-13T01:55:13.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:13.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj31 testdata.25784 2026-02-13T01:55:13.401 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:13.401 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj32 testdata.25784 2026-02-13T01:55:13.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:13.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj33 testdata.25784 2026-02-13T01:55:13.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:13.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj34 testdata.25784 2026-02-13T01:55:13.642 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:13.642 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj35 testdata.25784 2026-02-13T01:55:13.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:13.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj36 testdata.25784 2026-02-13T01:55:13.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:13.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj37 testdata.25784 2026-02-13T01:55:13.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:13.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj38 testdata.25784 2026-02-13T01:55:13.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:13.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj39 testdata.25784 2026-02-13T01:55:14.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:14.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj40 testdata.25784 2026-02-13T01:55:14.128 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:14.128 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj41 testdata.25784 2026-02-13T01:55:14.210 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:14.210 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj42 testdata.25784 2026-02-13T01:55:14.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:14.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj43 testdata.25784 2026-02-13T01:55:14.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:14.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj44 testdata.25784 2026-02-13T01:55:14.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:14.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj45 testdata.25784 2026-02-13T01:55:14.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:14.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj46 testdata.25784 2026-02-13T01:55:14.620 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:14.620 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj47 testdata.25784 2026-02-13T01:55:14.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:14.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj48 testdata.25784 2026-02-13T01:55:14.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:14.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj49 testdata.25784 2026-02-13T01:55:14.871 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:14.871 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj50 testdata.25784 2026-02-13T01:55:14.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:14.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj51 testdata.25784 2026-02-13T01:55:15.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:15.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj52 testdata.25784 2026-02-13T01:55:15.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:15.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj53 testdata.25784 2026-02-13T01:55:15.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:15.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj54 testdata.25784 2026-02-13T01:55:15.289 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:15.289 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj55 testdata.25784 2026-02-13T01:55:15.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:15.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj56 testdata.25784 2026-02-13T01:55:15.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:15.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj57 testdata.25784 2026-02-13T01:55:15.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:15.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj58 testdata.25784 2026-02-13T01:55:15.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:15.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj59 testdata.25784 2026-02-13T01:55:15.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:15.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj60 testdata.25784 2026-02-13T01:55:15.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:15.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj61 testdata.25784 2026-02-13T01:55:15.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:15.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj62 testdata.25784 2026-02-13T01:55:15.971 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:15.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj63 testdata.25784 2026-02-13T01:55:16.058 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:16.058 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj64 testdata.25784 2026-02-13T01:55:16.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:16.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj65 testdata.25784 2026-02-13T01:55:16.222 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:16.222 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj66 testdata.25784 2026-02-13T01:55:16.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:16.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj67 testdata.25784 2026-02-13T01:55:16.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:16.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj68 testdata.25784 2026-02-13T01:55:16.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:16.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj69 testdata.25784 2026-02-13T01:55:16.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:16.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj70 testdata.25784 2026-02-13T01:55:16.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:16.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj71 testdata.25784 2026-02-13T01:55:16.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:16.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj72 testdata.25784 2026-02-13T01:55:16.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:16.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj73 testdata.25784 2026-02-13T01:55:16.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:16.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj74 testdata.25784 2026-02-13T01:55:17.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:17.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj75 testdata.25784 2026-02-13T01:55:17.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:17.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj76 testdata.25784 2026-02-13T01:55:17.170 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:17.170 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj77 testdata.25784 2026-02-13T01:55:17.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:17.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj78 testdata.25784 2026-02-13T01:55:17.343 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:17.343 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj79 testdata.25784 2026-02-13T01:55:17.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:17.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj80 testdata.25784 2026-02-13T01:55:17.515 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:17.515 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj81 testdata.25784 2026-02-13T01:55:17.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:17.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj82 testdata.25784 2026-02-13T01:55:17.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:17.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj83 testdata.25784 2026-02-13T01:55:17.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:17.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj84 testdata.25784 2026-02-13T01:55:17.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:17.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj85 testdata.25784 2026-02-13T01:55:17.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:17.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj86 testdata.25784 2026-02-13T01:55:18.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:18.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj87 testdata.25784 2026-02-13T01:55:18.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:18.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj88 testdata.25784 2026-02-13T01:55:18.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:18.206 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj89 testdata.25784 2026-02-13T01:55:18.291 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:18.291 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj90 testdata.25784 2026-02-13T01:55:18.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:18.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj91 testdata.25784 2026-02-13T01:55:18.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:18.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj92 testdata.25784 2026-02-13T01:55:18.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:18.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj93 testdata.25784 2026-02-13T01:55:18.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:18.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj94 testdata.25784 2026-02-13T01:55:18.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:18.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj95 testdata.25784 2026-02-13T01:55:18.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:18.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj96 testdata.25784 2026-02-13T01:55:18.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:18.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj97 testdata.25784 2026-02-13T01:55:18.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:18.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj98 testdata.25784 2026-02-13T01:55:19.051 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:19.051 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj99 testdata.25784 2026-02-13T01:55:19.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-13T01:55:19.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj100 testdata.25784 2026-02-13T01:55:19.223 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:72: TEST_recovery_scrub_1: rm -f testdata.25784 2026-02-13T01:55:19.232 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:74: TEST_recovery_scrub_1: ceph osd pool set test size 4 2026-02-13T01:55:19.436 INFO:tasks.workunit.client.0.trial112.stderr:set pool 1 size to 4 2026-02-13T01:55:19.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:77: TEST_recovery_scrub_1: set -o pipefail 2026-02-13T01:55:19.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: count=0 2026-02-13T01:55:19.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2026-02-13T01:55:19.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2026-02-13T01:55:19.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2026-02-13T01:55:19.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2026-02-13T01:55:19.599 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T01:55:19.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2026-02-13T01:55:21.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 0 -eq 10 2026-02-13T01:55:21.609 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 0 + 1 2026-02-13T01:55:21.611 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=1 2026-02-13T01:55:21.611 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2026-02-13T01:55:21.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2026-02-13T01:55:21.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2026-02-13T01:55:21.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2026-02-13T01:55:21.760 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T01:55:21.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2026-02-13T01:55:23.774 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 1 -eq 10 2026-02-13T01:55:23.775 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 1 + 1 2026-02-13T01:55:23.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=2 2026-02-13T01:55:23.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2026-02-13T01:55:23.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2026-02-13T01:55:23.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2026-02-13T01:55:23.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2026-02-13T01:55:23.943 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T01:55:23.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:84: TEST_recovery_scrub_1: break 2026-02-13T01:55:23.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:94: TEST_recovery_scrub_1: set +o pipefail 2026-02-13T01:55:23.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:95: TEST_recovery_scrub_1: ceph pg dump pgs 2026-02-13T01:55:24.106 INFO:tasks.workunit.client.0.trial112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-13T01:55:24.107 INFO:tasks.workunit.client.0.trial112.stdout:1.0 100 0 291 0 0 5242880000 0 0 1300 0 1300 active+recovering+undersized+degraded+remapped 2026-02-13T01:55:19.920041+0000 20'1300 23:1335 [1,0,2,3] 1 [1,0] 1 0'0 2026-02-13T01:55:05.354036+0000 0'0 2026-02-13T01:55:05.354036+0000 0 0 no scrub is scheduled 0 0 2026-02-13T01:55:24.107 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:55:24.107 INFO:tasks.workunit.client.0.trial112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-13T01:55:24.107 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T01:55:24.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:97: TEST_recovery_scrub_1: sleep 10 2026-02-13T01:55:34.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:99: TEST_recovery_scrub_1: kill_daemons td/osd-recovery-scrub 2026-02-13T01:55:34.118 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T01:55:34.118 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T01:55:34.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T01:55:34.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T01:55:34.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T01:55:39.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T01:55:39.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:101: TEST_recovery_scrub_1: declare -a err_strings 2026-02-13T01:55:39.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub_1: err_strings[0]='recovery in progress.*scrubs' 2026-02-13T01:55:39.433 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: expr 4 - 1 2026-02-13T01:55:39.434 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: seq 0 3 2026-02-13T01:55:39.435 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T01:55:39.435 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.0.log 2026-02-13T01:55:39.439 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:20.150+0000 7fe3c99ee640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.439 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:21.138+0000 7fe3c99ee640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.439 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:22.178+0000 7fe3c99ee640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.440 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:23.158+0000 7fe3c99ee640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.440 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:24.150+0000 7fe3c99ee640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.440 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:25.138+0000 7fe3c99ee640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.440 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:26.138+0000 7fe3c99ee640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.440 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:27.138+0000 7fe3c99ee640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.440 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:28.154+0000 7fe3c99ee640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.440 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:29.190+0000 7fe3c99ee640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.440 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:30.202+0000 7fe3c99ee640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.440 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:31.191+0000 7fe3c99ee640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.440 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:32.175+0000 7fe3c99ee640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.440 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:33.183+0000 7fe3c99ee640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.441 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:34.215+0000 7fe3c99ee640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.441 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:35.243+0000 7fe3c99ee640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.441 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:36.219+0000 7fe3c99ee640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.441 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:37.179+0000 7fe3c99ee640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.441 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:38.179+0000 7fe3c99ee640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.441 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T01:55:39.441 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.1.log 2026-02-13T01:55:39.454 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:20.694+0000 7fe5a6dba640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.454 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:21.702+0000 7fe5a6dba640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.454 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:22.722+0000 7fe5a6dba640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.454 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:23.742+0000 7fe5a6dba640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.455 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:24.714+0000 7fe5a6dba640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.455 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:25.718+0000 7fe5a6dba640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.455 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:26.706+0000 7fe5a6dba640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.455 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:27.666+0000 7fe5a6dba640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.455 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:28.698+0000 7fe5a6dba640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.455 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:29.738+0000 7fe5a6dba640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.455 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:30.699+0000 7fe5a6dba640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.455 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:31.739+0000 7fe5a6dba640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.455 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:32.747+0000 7fe5a6dba640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.455 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:33.783+0000 7fe5a6dba640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.455 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:34.755+0000 7fe5a6dba640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.456 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:35.799+0000 7fe5a6dba640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.456 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:36.755+0000 7fe5a6dba640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.456 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:37.767+0000 7fe5a6dba640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.456 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:38.763+0000 7fe5a6dba640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.456 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T01:55:39.456 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.2.log 2026-02-13T01:55:39.458 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:20.894+0000 7f16449ab640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.458 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:21.918+0000 7f16449ab640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.458 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:22.878+0000 7f16449ab640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.458 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:23.850+0000 7f16449ab640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.458 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:24.814+0000 7f16449ab640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.459 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:25.826+0000 7f16449ab640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.459 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:26.826+0000 7f16449ab640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.459 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:27.866+0000 7f16449ab640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.459 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:28.822+0000 7f16449ab640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.459 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:29.846+0000 7f16449ab640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.459 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:30.895+0000 7f16449ab640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.459 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:31.895+0000 7f16449ab640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.459 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:32.927+0000 7f16449ab640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.459 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:33.887+0000 7f16449ab640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.459 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:34.919+0000 7f16449ab640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.459 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:35.895+0000 7f16449ab640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.459 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:36.859+0000 7f16449ab640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.460 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:37.875+0000 7f16449ab640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.460 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:38.883+0000 7f16449ab640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T01:55:39.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.3.log 2026-02-13T01:55:39.462 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:20.662+0000 7f0aea455640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.462 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:21.658+0000 7f0aea455640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.462 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:22.654+0000 7f0aea455640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.462 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:23.614+0000 7f0aea455640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.462 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:24.642+0000 7f0aea455640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.463 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:25.662+0000 7f0aea455640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.463 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:26.626+0000 7f0aea455640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.463 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:27.670+0000 7f0aea455640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.463 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:28.670+0000 7f0aea455640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.463 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:29.682+0000 7f0aea455640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.463 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:30.691+0000 7f0aea455640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.463 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:31.667+0000 7f0aea455640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.463 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:32.687+0000 7f0aea455640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.463 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:33.647+0000 7f0aea455640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.463 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:34.647+0000 7f0aea455640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.463 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:35.611+0000 7f0aea455640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.463 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:36.579+0000 7f0aea455640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.464 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:37.575+0000 7f0aea455640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.464 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:55:38.575+0000 7f0aea455640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-13T01:55:39.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: for err_string in "${err_strings[@]}" 2026-02-13T01:55:39.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:110: TEST_recovery_scrub_1: found=false 2026-02-13T01:55:39.465 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:111: TEST_recovery_scrub_1: count=0 2026-02-13T01:55:39.465 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: expr 4 - 1 2026-02-13T01:55:39.465 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: seq 0 3 2026-02-13T01:55:39.465 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T01:55:39.465 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.0.log 2026-02-13T01:55:39.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2026-02-13T01:55:39.467 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 0 + 1 2026-02-13T01:55:39.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=1 2026-02-13T01:55:39.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T01:55:39.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.1.log 2026-02-13T01:55:39.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2026-02-13T01:55:39.477 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 1 + 1 2026-02-13T01:55:39.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=2 2026-02-13T01:55:39.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T01:55:39.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.2.log 2026-02-13T01:55:39.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2026-02-13T01:55:39.480 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 2 + 1 2026-02-13T01:55:39.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=3 2026-02-13T01:55:39.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T01:55:39.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.3.log 2026-02-13T01:55:39.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2026-02-13T01:55:39.483 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 3 + 1 2026-02-13T01:55:39.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=4 2026-02-13T01:55:39.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:120: TEST_recovery_scrub_1: '[' true = false ']' 2026-02-13T01:55:39.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:124: TEST_recovery_scrub_1: '[' 4 -eq 4 ']' 2026-02-13T01:55:39.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:127: TEST_recovery_scrub_1: teardown td/osd-recovery-scrub 2026-02-13T01:55:39.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-recovery-scrub 2026-02-13T01:55:39.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T01:55:39.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-recovery-scrub KILL 2026-02-13T01:55:39.485 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T01:55:39.485 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T01:55:39.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T01:55:39.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T01:55:39.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T01:55:39.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T01:55:39.490 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T01:55:39.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T01:55:39.491 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T01:55:39.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T01:55:39.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T01:55:39.493 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T01:55:39.494 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T01:55:39.494 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T01:55:39.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T01:55:39.495 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T01:55:39.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T01:55:39.497 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T01:55:39.497 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-recovery-scrub 2026-02-13T01:55:39.542 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T01:55:39.542 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:55:39.542 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.25784 2026-02-13T01:55:39.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.25784 2026-02-13T01:55:39.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T01:55:39.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T01:55:39.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:129: TEST_recovery_scrub_1: '[' 0 '!=' 0 ']' 2026-02-13T01:55:39.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:135: TEST_recovery_scrub_1: echo 'TEST PASSED' 2026-02-13T01:55:39.543 INFO:tasks.workunit.client.0.trial112.stdout:TEST PASSED 2026-02-13T01:55:39.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:136: TEST_recovery_scrub_1: return 0 2026-02-13T01:55:39.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: main: code=0 2026-02-13T01:55:39.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: teardown td/osd-recovery-scrub 0 2026-02-13T01:55:39.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-recovery-scrub 2026-02-13T01:55:39.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-13T01:55:39.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-recovery-scrub KILL 2026-02-13T01:55:39.545 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T01:55:39.545 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T01:55:39.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T01:55:39.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T01:55:39.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T01:55:39.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T01:55:39.546 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T01:55:39.547 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T01:55:39.547 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T01:55:39.548 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T01:55:39.548 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T01:55:39.548 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T01:55:39.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T01:55:39.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T01:55:39.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T01:55:39.550 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T01:55:39.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T01:55:39.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-13T01:55:39.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-recovery-scrub 2026-02-13T01:55:39.553 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T01:55:39.553 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:55:39.553 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.25784 2026-02-13T01:55:39.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.25784 2026-02-13T01:55:39.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T01:55:39.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T01:55:39.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: main: return 0 2026-02-13T01:55:39.556 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-13T01:55:39.556 DEBUG:teuthology.orchestra.run.trial112:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-13T01:55:39.610 INFO:tasks.workunit:Running workunit scrub/osd-scrub-dump.sh... 2026-02-13T01:55:39.610 DEBUG:teuthology.orchestra.run.trial112:workunit test scrub/osd-scrub-dump.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh 2026-02-13T01:55:39.663 INFO:tasks.workunit.client.0.trial112.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-13T01:55:39.666 INFO:tasks.workunit.client.0.trial112.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-13T01:55:39.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-13T01:55:39.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-13T01:55:39.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export PYTHONWARNINGS=ignore 2026-02-13T01:55:39.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: PYTHONWARNINGS=ignore 2026-02-13T01:55:39.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: export CEPH_CONF=/dev/null 2026-02-13T01:55:39.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: CEPH_CONF=/dev/null 2026-02-13T01:55:39.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: main: unset CEPH_ARGS 2026-02-13T01:55:39.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: local code 2026-02-13T01:55:39.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: run td/osd-scrub-dump 2026-02-13T01:55:39.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:30: run: echo 'This test is disabled' 2026-02-13T01:55:39.667 INFO:tasks.workunit.client.0.trial112.stdout:This test is disabled 2026-02-13T01:55:39.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:31: run: return 0 2026-02-13T01:55:39.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: main: code=0 2026-02-13T01:55:39.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: teardown td/osd-scrub-dump 0 2026-02-13T01:55:39.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-dump 2026-02-13T01:55:39.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-13T01:55:39.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-dump KILL 2026-02-13T01:55:39.669 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T01:55:39.669 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T01:55:39.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T01:55:39.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T01:55:39.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T01:55:39.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T01:55:39.670 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T01:55:39.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T01:55:39.670 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T01:55:39.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T01:55:39.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T01:55:39.672 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T01:55:39.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T01:55:39.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T01:55:39.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T01:55:39.673 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T01:55:39.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T01:55:39.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-13T01:55:39.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-dump 2026-02-13T01:55:39.677 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T01:55:39.677 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:55:39.677 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35183 2026-02-13T01:55:39.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35183 2026-02-13T01:55:39.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T01:55:39.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T01:55:39.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: main: return 0 2026-02-13T01:55:39.680 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-13T01:55:39.680 DEBUG:teuthology.orchestra.run.trial112:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-13T01:55:39.730 INFO:tasks.workunit:Running workunit scrub/osd-scrub-repair.sh... 2026-02-13T01:55:39.730 DEBUG:teuthology.orchestra.run.trial112:workunit test scrub/osd-scrub-repair.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh 2026-02-13T01:55:39.785 INFO:tasks.workunit.client.0.trial112.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh 2026-02-13T01:55:39.785 INFO:tasks.workunit.client.0.trial112.stderr:++ TIMEOUT=300 2026-02-13T01:55:39.785 INFO:tasks.workunit.client.0.trial112.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2026-02-13T01:55:39.785 INFO:tasks.workunit.client.0.trial112.stderr:++ MAX_TIMEOUT=15 2026-02-13T01:55:39.785 INFO:tasks.workunit.client.0.trial112.stderr:++ PG_NUM=4 2026-02-13T01:55:39.785 INFO:tasks.workunit.client.0.trial112.stderr:++ TMPDIR=/tmp 2026-02-13T01:55:39.786 INFO:tasks.workunit.client.0.trial112.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2026-02-13T01:55:39.786 INFO:tasks.workunit.client.0.trial112.stderr:++ TESTDIR=/home/ubuntu/cephtest 2026-02-13T01:55:39.786 INFO:tasks.workunit.client.0.trial112.stderr:+++ uname 2026-02-13T01:55:39.787 INFO:tasks.workunit.client.0.trial112.stderr:++ '[' Linux = FreeBSD ']' 2026-02-13T01:55:39.787 INFO:tasks.workunit.client.0.trial112.stderr:++ SED=sed 2026-02-13T01:55:39.787 INFO:tasks.workunit.client.0.trial112.stderr:++ AWK=awk 2026-02-13T01:55:39.787 INFO:tasks.workunit.client.0.trial112.stderr:+++ stty -a 2026-02-13T01:55:39.787 INFO:tasks.workunit.client.0.trial112.stderr:+++ head -1 2026-02-13T01:55:39.787 INFO:tasks.workunit.client.0.trial112.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2026-02-13T01:55:39.787 INFO:tasks.workunit.client.0.trial112.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-13T01:55:39.788 INFO:tasks.workunit.client.0.trial112.stderr:++ termwidth= 2026-02-13T01:55:39.788 INFO:tasks.workunit.client.0.trial112.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2026-02-13T01:55:39.788 INFO:tasks.workunit.client.0.trial112.stderr:++ DIFFCOLOPTS='-y ' 2026-02-13T01:55:39.789 INFO:tasks.workunit.client.0.trial112.stderr:++ KERNCORE=kernel.core_pattern 2026-02-13T01:55:39.789 INFO:tasks.workunit.client.0.trial112.stderr:++ EXTRA_OPTS= 2026-02-13T01:55:39.791 INFO:tasks.workunit.client.0.trial112.stderr:++ test '' = TESTS 2026-02-13T01:55:39.791 INFO:tasks.workunit.client.0.trial112.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh 2026-02-13T01:55:39.792 INFO:tasks.workunit.client.0.trial112.stderr:++ uname 2026-02-13T01:55:39.793 INFO:tasks.workunit.client.0.trial112.stderr:+ '[' Linux = FreeBSD ']' 2026-02-13T01:55:39.793 INFO:tasks.workunit.client.0.trial112.stderr:+ use_ec_overwrite=true 2026-02-13T01:55:39.793 INFO:tasks.workunit.client.0.trial112.stderr:+ getjson=no 2026-02-13T01:55:39.793 INFO:tasks.workunit.client.0.trial112.stderr:+ jqfilter='def walk(f): 2026-02-13T01:55:39.794 INFO:tasks.workunit.client.0.trial112.stderr: . as $in 2026-02-13T01:55:39.794 INFO:tasks.workunit.client.0.trial112.stderr: | if type == "object" then 2026-02-13T01:55:39.794 INFO:tasks.workunit.client.0.trial112.stderr: reduce keys[] as $key 2026-02-13T01:55:39.794 INFO:tasks.workunit.client.0.trial112.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-13T01:55:39.794 INFO:tasks.workunit.client.0.trial112.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-13T01:55:39.794 INFO:tasks.workunit.client.0.trial112.stderr: else f 2026-02-13T01:55:39.794 INFO:tasks.workunit.client.0.trial112.stderr: end; 2026-02-13T01:55:39.794 INFO:tasks.workunit.client.0.trial112.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-13T01:55:39.794 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-13T01:55:39.794 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-13T01:55:39.794 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-13T01:55:39.794 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-13T01:55:39.795 INFO:tasks.workunit.client.0.trial112.stderr:+ sortkeys='import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-13T01:55:39.799 INFO:tasks.workunit.client.0.trial112.stderr:+ main osd-scrub-repair 2026-02-13T01:55:39.799 INFO:tasks.workunit.client.0.trial112.stderr:+ local dir=td/osd-scrub-repair 2026-02-13T01:55:39.799 INFO:tasks.workunit.client.0.trial112.stderr:+ shift 2026-02-13T01:55:39.799 INFO:tasks.workunit.client.0.trial112.stderr:+ shopt -s -o xtrace 2026-02-13T01:55:39.799 INFO:tasks.workunit.client.0.trial112.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-13T01:55:39.799 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-13T01:55:39.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-13T01:55:39.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export PYTHONWARNINGS=ignore 2026-02-13T01:55:39.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: PYTHONWARNINGS=ignore 2026-02-13T01:55:39.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: export CEPH_CONF=/dev/null 2026-02-13T01:55:39.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: CEPH_CONF=/dev/null 2026-02-13T01:55:39.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: main: unset CEPH_ARGS 2026-02-13T01:55:39.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: local code 2026-02-13T01:55:39.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: run td/osd-scrub-repair 2026-02-13T01:55:39.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:52: run: local dir=td/osd-scrub-repair 2026-02-13T01:55:39.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:53: run: shift 2026-02-13T01:55:39.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: export CEPH_MON=127.0.0.1:7107 2026-02-13T01:55:39.801 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: CEPH_MON=127.0.0.1:7107 2026-02-13T01:55:39.801 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:56: run: export CEPH_ARGS 2026-02-13T01:55:39.801 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: uuidgen 2026-02-13T01:55:39.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: CEPH_ARGS+='--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none ' 2026-02-13T01:55:39.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:58: run: CEPH_ARGS+='--mon-host=127.0.0.1:7107 ' 2026-02-13T01:55:39.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:59: run: CEPH_ARGS+='--osd-skip-data-digest=false ' 2026-02-13T01:55:39.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:61: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2026-02-13T01:55:39.802 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: set 2026-02-13T01:55:39.802 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-13T01:55:39.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: local 'funcs=TEST_allow_oper_initiated_scrub_during_recovery 2026-02-13T01:55:39.805 INFO:tasks.workunit.client.0.trial112.stderr:TEST_allow_repair_during_recovery 2026-02-13T01:55:39.805 INFO:tasks.workunit.client.0.trial112.stderr:TEST_auto_repair_bluestore_basic 2026-02-13T01:55:39.805 INFO:tasks.workunit.client.0.trial112.stderr:TEST_auto_repair_bluestore_failed 2026-02-13T01:55:39.805 INFO:tasks.workunit.client.0.trial112.stderr:TEST_auto_repair_bluestore_failed_norecov 2026-02-13T01:55:39.805 INFO:tasks.workunit.client.0.trial112.stderr:TEST_auto_repair_bluestore_scrub 2026-02-13T01:55:39.805 INFO:tasks.workunit.client.0.trial112.stderr:TEST_auto_repair_bluestore_tag 2026-02-13T01:55:39.805 INFO:tasks.workunit.client.0.trial112.stderr:TEST_auto_repair_erasure_coded_appends 2026-02-13T01:55:39.805 INFO:tasks.workunit.client.0.trial112.stderr:TEST_auto_repair_erasure_coded_overwrites 2026-02-13T01:55:39.805 INFO:tasks.workunit.client.0.trial112.stderr:TEST_corrupt_and_repair_jerasure_appends 2026-02-13T01:55:39.806 INFO:tasks.workunit.client.0.trial112.stderr:TEST_corrupt_and_repair_jerasure_overwrites 2026-02-13T01:55:39.806 INFO:tasks.workunit.client.0.trial112.stderr:TEST_corrupt_and_repair_lrc_appends 2026-02-13T01:55:39.806 INFO:tasks.workunit.client.0.trial112.stderr:TEST_corrupt_and_repair_lrc_overwrites 2026-02-13T01:55:39.806 INFO:tasks.workunit.client.0.trial112.stderr:TEST_corrupt_and_repair_replicated 2026-02-13T01:55:39.806 INFO:tasks.workunit.client.0.trial112.stderr:TEST_corrupt_scrub_erasure_appends 2026-02-13T01:55:39.806 INFO:tasks.workunit.client.0.trial112.stderr:TEST_corrupt_scrub_erasure_overwrites 2026-02-13T01:55:39.806 INFO:tasks.workunit.client.0.trial112.stderr:TEST_corrupt_scrub_replicated 2026-02-13T01:55:39.806 INFO:tasks.workunit.client.0.trial112.stderr:TEST_corrupt_snapset_scrub_rep 2026-02-13T01:55:39.806 INFO:tasks.workunit.client.0.trial112.stderr:TEST_dual_store_replicated_cluster 2026-02-13T01:55:39.806 INFO:tasks.workunit.client.0.trial112.stderr:TEST_list_missing_erasure_coded_appends 2026-02-13T01:55:39.806 INFO:tasks.workunit.client.0.trial112.stderr:TEST_list_missing_erasure_coded_overwrites 2026-02-13T01:55:39.807 INFO:tasks.workunit.client.0.trial112.stderr:TEST_periodic_scrub_replicated 2026-02-13T01:55:39.807 INFO:tasks.workunit.client.0.trial112.stderr:TEST_repair_stats 2026-02-13T01:55:39.807 INFO:tasks.workunit.client.0.trial112.stderr:TEST_repair_stats_ec 2026-02-13T01:55:39.807 INFO:tasks.workunit.client.0.trial112.stderr:TEST_request_scrub_priority 2026-02-13T01:55:39.807 INFO:tasks.workunit.client.0.trial112.stderr:TEST_scrub_warning 2026-02-13T01:55:39.807 INFO:tasks.workunit.client.0.trial112.stderr:TEST_skip_non_repair_during_recovery 2026-02-13T01:55:39.807 INFO:tasks.workunit.client.0.trial112.stderr:TEST_unfound_erasure_coded_appends 2026-02-13T01:55:39.807 INFO:tasks.workunit.client.0.trial112.stderr:TEST_unfound_erasure_coded_overwrites' 2026-02-13T01:55:39.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-13T01:55:39.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-13T01:55:39.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-13T01:55:39.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-13T01:55:39.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T01:55:39.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T01:55:39.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T01:55:39.808 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T01:55:39.808 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T01:55:39.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T01:55:39.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T01:55:39.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T01:55:39.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T01:55:39.809 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T01:55:39.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T01:55:39.809 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T01:55:39.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T01:55:39.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T01:55:39.810 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T01:55:39.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T01:55:39.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T01:55:39.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T01:55:39.812 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T01:55:39.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T01:55:39.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T01:55:39.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T01:55:39.816 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T01:55:39.816 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:55:39.816 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:55:39.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T01:55:39.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T01:55:39.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T01:55:39.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-13T01:55:39.819 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T01:55:39.820 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:55:39.820 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:55:39.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.35207 2026-02-13T01:55:39.821 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T01:55:39.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-13T01:55:39.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-13T01:55:39.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T01:55:39.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-13T01:55:39.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_allow_oper_initiated_scrub_during_recovery td/osd-scrub-repair 2026-02-13T01:55:39.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:115: TEST_allow_oper_initiated_scrub_during_recovery: local dir=td/osd-scrub-repair 2026-02-13T01:55:39.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:116: TEST_allow_oper_initiated_scrub_during_recovery: cluster_conf=(['osds_num']='2' ['pgs_in_pool']='4' ['pool_name']='nopool' ['pool_default_size']='2' ['extras']='--osd_scrub_during_recovery=false --osd_debug_pretend_recovery_active=true') 2026-02-13T01:55:39.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:116: TEST_allow_oper_initiated_scrub_during_recovery: local -A cluster_conf 2026-02-13T01:55:39.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:125: TEST_allow_oper_initiated_scrub_during_recovery: standard_scrub_cluster td/osd-scrub-repair cluster_conf 2026-02-13T01:55:39.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local dir=td/osd-scrub-repair 2026-02-13T01:55:39.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:230: standard_scrub_cluster: local -n args=cluster_conf 2026-02-13T01:55:39.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local OSDS=2 2026-02-13T01:55:39.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: local pg_num=4 2026-02-13T01:55:39.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:235: standard_scrub_cluster: local poolname=nopool 2026-02-13T01:55:39.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:236: standard_scrub_cluster: args['pool_name']=nopool 2026-02-13T01:55:39.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:238: standard_scrub_cluster: local pool_default_size=2 2026-02-13T01:55:39.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:239: standard_scrub_cluster: args['pool_default_size']=2 2026-02-13T01:55:39.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:241: standard_scrub_cluster: local 'extra_pars=--osd_scrub_during_recovery=false --osd_debug_pretend_recovery_active=true' 2026-02-13T01:55:39.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:242: standard_scrub_cluster: local debug_msg=dbg 2026-02-13T01:55:39.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:245: standard_scrub_cluster: local saved_echo_flag=x 2026-02-13T01:55:39.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:246: standard_scrub_cluster: set +x 2026-02-13T01:55:40.111 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 a4fb40fe-098c-42f1-8527-dd2598830026 2026-02-13T01:55:40.212 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T01:55:40.243 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:40.235+0000 7f4736486a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:55:40.244 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:40.235+0000 7f4736486a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:55:40.245 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:40.235+0000 7f4736486a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:55:40.246 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:40.235+0000 7f4736486a40 -1 bdev(0x55810a88cc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-13T01:55:40.246 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:40.239+0000 7f4736486a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-13T01:55:41.014 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T01:55:41.116 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T01:55:41.149 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:41.139+0000 7fdcb62f9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:55:41.151 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:41.143+0000 7fdcb62f9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:55:41.152 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:41.143+0000 7fdcb62f9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:55:41.206 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T01:55:41.651 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:41.643+0000 7fdcb62f9a40 -1 Falling back to public interface 2026-02-13T01:55:42.299 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T01:55:42.555 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:42.547+0000 7fdcb62f9a40 -1 osd.0 0 log_to_monitors true 2026-02-13T01:55:43.389 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T01:55:43.548 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1953472276,v1:127.0.0.1:6801/1953472276] [v2:127.0.0.1:6802/1953472276,v1:127.0.0.1:6803/1953472276] exists,up a4fb40fe-098c-42f1-8527-dd2598830026 2026-02-13T01:55:43.552 INFO:tasks.workunit.client.0.trial112.stdout:add osd1 489ebe61-84b9-4b0c-b1ba-1156f1fb2567 2026-02-13T01:55:43.729 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T01:55:43.759 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:43.751+0000 7f8807521a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:55:43.761 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:43.751+0000 7f8807521a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:55:43.762 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:43.755+0000 7f8807521a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:55:43.763 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:43.755+0000 7f8807521a40 -1 bdev(0x5592ae85ac00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-13T01:55:43.763 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:43.755+0000 7f8807521a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-13T01:55:44.450 INFO:tasks.workunit.client.0.trial112.stdout:adding osd1 key to auth repository 2026-02-13T01:55:44.663 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T01:55:44.694 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:44.687+0000 7f32ec12da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:55:44.696 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:44.687+0000 7f32ec12da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:55:44.697 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:44.687+0000 7f32ec12da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:55:44.819 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T01:55:45.063 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:45.056+0000 7f32ec12da40 -1 Falling back to public interface 2026-02-13T01:55:45.644 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:45.636+0000 7f32ec12da40 -1 osd.1 0 log_to_monitors true 2026-02-13T01:55:45.986 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T01:55:47.145 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T01:55:47.305 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/37636691,v1:127.0.0.1:6811/37636691] [v2:127.0.0.1:6812/37636691,v1:127.0.0.1:6813/37636691] exists,up 489ebe61-84b9-4b0c-b1ba-1156f1fb2567 2026-02-13T01:55:48.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:126: TEST_allow_oper_initiated_scrub_during_recovery: local poolname=rbd 2026-02-13T01:55:48.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:127: TEST_allow_oper_initiated_scrub_during_recovery: create_rbd_pool 2026-02-13T01:55:48.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-13T01:55:48.455 INFO:tasks.workunit.client.0.trial112.stderr:pool 'rbd' does not exist 2026-02-13T01:55:48.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-13T01:55:48.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-13T01:55:48.679 INFO:tasks.workunit.client.0.trial112.stderr:pool 'rbd' created 2026-02-13T01:55:48.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T01:55:49.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-13T01:55:49.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:128: TEST_allow_oper_initiated_scrub_during_recovery: wait_for_clean 2026-02-13T01:55:49.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T01:55:49.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T01:55:49.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T01:55:49.974 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T01:55:49.974 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T01:55:49.974 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T01:55:49.975 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T01:55:49.975 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T01:55:49.975 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T01:55:50.082 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T01:55:50.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T01:55:50.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T01:55:50.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T01:55:50.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T01:55:50.083 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T01:55:50.243 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T01:55:50.243 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T01:55:50.243 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T01:55:50.243 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:55:50.244 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T01:55:50.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836488 2026-02-13T01:55:50.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836488 2026-02-13T01:55:50.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488' 2026-02-13T01:55:50.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:55:50.317 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T01:55:50.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672965 2026-02-13T01:55:50.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672965 2026-02-13T01:55:50.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-42949672965' 2026-02-13T01:55:50.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:55:50.389 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836488 2026-02-13T01:55:50.389 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:55:50.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T01:55:50.391 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836488 2026-02-13T01:55:50.391 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:55:50.393 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836488 2026-02-13T01:55:50.393 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2026-02-13T01:55:50.393 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836488 2026-02-13T01:55:50.393 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:55:50.568 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836488 2026-02-13T01:55:50.568 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:55:50.569 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672965 2026-02-13T01:55:50.569 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:55:50.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T01:55:50.571 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672965 2026-02-13T01:55:50.571 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:55:50.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672965 2026-02-13T01:55:50.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2026-02-13T01:55:50.573 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 42949672965 2026-02-13T01:55:50.573 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T01:55:50.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672964 -lt 42949672965 2026-02-13T01:55:50.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:55:51.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T01:55:51.744 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T01:55:51.905 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672964 -lt 42949672965 2026-02-13T01:55:51.905 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:55:52.906 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T01:55:52.907 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T01:55:53.074 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672967 -lt 42949672965 2026-02-13T01:55:53.074 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T01:55:53.075 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:55:53.075 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:55:53.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-13T01:55:53.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T01:55:53.269 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T01:55:53.269 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T01:55:53.269 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T01:55:53.269 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T01:55:53.269 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T01:55:53.270 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T01:55:53.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-13T01:55:53.425 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T01:55:53.425 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:55:53.426 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:55:53.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-13T01:55:53.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T01:55:53.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T01:55:53.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:130: TEST_allow_oper_initiated_scrub_during_recovery: add_something td/osd-scrub-repair rbd 2026-02-13T01:55:53.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T01:55:53.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2026-02-13T01:55:53.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2026-02-13T01:55:53.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T01:55:53.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T01:55:53.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T01:55:53.825 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T01:55:53.836 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T01:55:54.030 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T01:55:54.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T01:55:54.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T01:55:54.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-13T01:55:54.070 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:131: TEST_allow_oper_initiated_scrub_during_recovery: get_not_primary rbd SOMETHING 2026-02-13T01:55:54.070 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=rbd 2026-02-13T01:55:54.070 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-13T01:55:54.070 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary rbd SOMETHING 2026-02-13T01:55:54.070 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2026-02-13T01:55:54.070 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-13T01:55:54.071 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd SOMETHING 2026-02-13T01:55:54.071 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T01:55:54.226 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-13T01:55:54.226 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map rbd SOMETHING 2026-02-13T01:55:54.226 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-13T01:55:54.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:131: TEST_allow_oper_initiated_scrub_during_recovery: oper_scrub_and_schedule td/osd-scrub-repair rbd 0 2026-02-13T01:55:54.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:180: oper_scrub_and_schedule: local dir=td/osd-scrub-repair 2026-02-13T01:55:54.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:181: oper_scrub_and_schedule: local poolname=rbd 2026-02-13T01:55:54.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:182: oper_scrub_and_schedule: local osd=0 2026-02-13T01:55:54.391 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: oper_scrub_and_schedule: get_pg rbd SOMETHING 2026-02-13T01:55:54.392 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=rbd 2026-02-13T01:55:54.392 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-13T01:55:54.392 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map rbd SOMETHING 2026-02-13T01:55:54.392 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T01:55:54.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: oper_scrub_and_schedule: local pg=1.3 2026-02-13T01:55:54.554 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: oper_scrub_and_schedule: get_last_scrub_stamp 1.3 2026-02-13T01:55:54.554 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-13T01:55:54.554 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T01:55:54.554 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T01:55:54.554 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-13T01:55:54.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: oper_scrub_and_schedule: local last_scrub=2026-02-13T01:55:48.676057+0000 2026-02-13T01:55:54.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:189: oper_scrub_and_schedule: ceph tell 1.3 scrub 2026-02-13T01:55:54.781 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T01:55:54.781 INFO:tasks.workunit.client.0.trial112.stdout: "deep": false, 2026-02-13T01:55:54.781 INFO:tasks.workunit.client.0.trial112.stdout: "must": true, 2026-02-13T01:55:54.781 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "0.000000" 2026-02-13T01:55:54.781 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T01:55:54.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: oper_scrub_and_schedule: sleep 3 2026-02-13T01:55:57.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: oper_scrub_and_schedule: (( i=0 )) 2026-02-13T01:55:57.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: oper_scrub_and_schedule: (( i < 3 )) 2026-02-13T01:55:57.790 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: oper_scrub_and_schedule: get_last_scrub_stamp 1.3 2026-02-13T01:55:57.790 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-13T01:55:57.790 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T01:55:57.791 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T01:55:57.791 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-13T01:55:57.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: oper_scrub_and_schedule: test 2026-02-13T01:55:55.717157+0000 '>' 2026-02-13T01:55:48.676057+0000 2026-02-13T01:55:57.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: oper_scrub_and_schedule: break 2026-02-13T01:55:57.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: oper_scrub_and_schedule: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-13T01:55:57.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T01:55:57.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T01:55:57.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T01:55:57.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T01:55:57.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-13T01:55:57.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T01:55:57.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T01:55:57.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T01:55:57.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T01:55:57.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T01:55:57.955 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T01:55:57.955 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T01:55:57.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T01:55:57.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T01:55:57.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T01:55:58.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T01:55:58.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-13T01:55:58.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T01:55:58.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T01:55:58.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T01:55:58.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T01:55:58.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T01:55:58.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2026-02-13T01:55:58.351 INFO:tasks.workunit.client.0.trial112.stdout:_ 2026-02-13T01:55:58.351 INFO:tasks.workunit.client.0.trial112.stdout:snapset 2026-02-13T01:55:58.456 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-13T01:55:58.456 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T01:55:58.456 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T01:55:58.456 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T01:55:58.456 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T01:55:58.456 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T01:55:58.456 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T01:55:58.456 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T01:55:58.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T01:55:58.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T01:55:58.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T01:55:58.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T01:55:58.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T01:55:58.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T01:55:58.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T01:55:58.457 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T01:55:58.457 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:55:58.457 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:55:58.457 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:55:58.458 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:55:58.458 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:55:58.458 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:55:58.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:55:58.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T01:55:58.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T01:55:58.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T01:55:58.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T01:55:58.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T01:55:58.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T01:55:58.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T01:55:58.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T01:55:58.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T01:55:58.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T01:55:58.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T01:55:58.460 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T01:55:58.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T01:55:58.460 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T01:55:58.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T01:55:58.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T01:55:58.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T01:55:58.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T01:55:58.474 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:58.468+0000 7fc166b8fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:55:58.475 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:58.468+0000 7fc166b8fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:55:58.476 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:58.468+0000 7fc166b8fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:55:58.621 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T01:55:58.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T01:55:58.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T01:55:58.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T01:55:58.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T01:55:58.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:55:58.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T01:55:58.622 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T01:55:58.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:55:58.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T01:55:58.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:55:58.875 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:58.868+0000 7fc166b8fa40 -1 Falling back to public interface 2026-02-13T01:55:59.498 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:55:59.492+0000 7fc166b8fa40 -1 osd.0 20 log_to_monitors true 2026-02-13T01:55:59.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:55:59.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:55:59.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T01:55:59.785 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:55:59.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:55:59.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T01:55:59.941 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 23 up_thru 23 down_at 20 last_clean_interval [5,20) [v2:127.0.0.1:6800/828888062,v1:127.0.0.1:6801/828888062] [v2:127.0.0.1:6802/828888062,v1:127.0.0.1:6803/828888062] exists,up a4fb40fe-098c-42f1-8527-dd2598830026 2026-02-13T01:55:59.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T01:55:59.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T01:55:59.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T01:55:59.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T01:55:59.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T01:55:59.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T01:55:59.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T01:55:59.942 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T01:55:59.943 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T01:55:59.943 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T01:55:59.943 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T01:55:59.943 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T01:55:59.943 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T01:56:00.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T01:56:00.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T01:56:00.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T01:56:00.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T01:56:00.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T01:56:00.042 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T01:56:00.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T01:56:00.205 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T01:56:00.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T01:56:00.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:56:00.205 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T01:56:00.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247810 2026-02-13T01:56:00.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247810 2026-02-13T01:56:00.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247810' 2026-02-13T01:56:00.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:56:00.278 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T01:56:00.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672976 2026-02-13T01:56:00.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672976 2026-02-13T01:56:00.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247810 1-42949672976' 2026-02-13T01:56:00.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:56:00.351 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-98784247810 2026-02-13T01:56:00.352 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:56:00.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T01:56:00.353 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-98784247810 2026-02-13T01:56:00.353 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:56:00.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247810 2026-02-13T01:56:00.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 98784247810' 2026-02-13T01:56:00.354 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 98784247810 2026-02-13T01:56:00.355 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:56:00.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247810 -lt 98784247810 2026-02-13T01:56:00.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:56:00.518 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672976 2026-02-13T01:56:00.518 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:56:00.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T01:56:00.520 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672976 2026-02-13T01:56:00.520 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:56:00.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672976 2026-02-13T01:56:00.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2026-02-13T01:56:00.522 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 42949672976 2026-02-13T01:56:00.522 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T01:56:00.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672976 -lt 42949672976 2026-02-13T01:56:00.684 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T01:56:00.684 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:56:00.684 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:56:00.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-13T01:56:00.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T01:56:00.885 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T01:56:00.885 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T01:56:00.885 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T01:56:00.885 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T01:56:00.885 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T01:56:00.885 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T01:56:01.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-13T01:56:01.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T01:56:01.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:56:01.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:56:01.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-13T01:56:01.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T01:56:01.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T01:56:01.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: oper_scrub_and_schedule: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2026-02-13T01:56:01.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: oper_scrub_and_schedule: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-13T01:56:01.267 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-13T01:56:01.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T01:56:01.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T01:56:01.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T01:56:01.269 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T01:56:01.269 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T01:56:01.269 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T01:56:01.269 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T01:56:01.269 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T01:56:01.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T01:56:01.378 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T01:56:01.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T01:56:01.379 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T01:56:01.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T01:56:01.381 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T01:56:01.381 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T01:56:01.382 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T01:56:01.382 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T01:56:01.383 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T01:56:01.383 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T01:56:01.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T01:56:01.385 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T01:56:01.385 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T01:56:01.393 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T01:56:01.393 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:56:01.393 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:56:01.393 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T01:56:01.395 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T01:56:01.395 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T01:56:01.395 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-13T01:56:01.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-13T01:56:01.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-13T01:56:01.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-13T01:56:01.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T01:56:01.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T01:56:01.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T01:56:01.396 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T01:56:01.396 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T01:56:01.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T01:56:01.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T01:56:01.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T01:56:01.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T01:56:01.398 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T01:56:01.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T01:56:01.399 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T01:56:01.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T01:56:01.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T01:56:01.401 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T01:56:01.402 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T01:56:01.402 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T01:56:01.402 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T01:56:01.402 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T01:56:01.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T01:56:01.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T01:56:01.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T01:56:01.406 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T01:56:01.406 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:56:01.406 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:56:01.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T01:56:01.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T01:56:01.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T01:56:01.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-13T01:56:01.409 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T01:56:01.410 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:56:01.410 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:56:01.410 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.35207 2026-02-13T01:56:01.411 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T01:56:01.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-13T01:56:01.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T01:56:01.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-13T01:56:01.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_allow_repair_during_recovery td/osd-scrub-repair 2026-02-13T01:56:01.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:138: TEST_allow_repair_during_recovery: local dir=td/osd-scrub-repair 2026-02-13T01:56:01.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:139: TEST_allow_repair_during_recovery: local poolname=rbd 2026-02-13T01:56:01.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:141: TEST_allow_repair_during_recovery: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2026-02-13T01:56:01.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-13T01:56:01.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-13T01:56:01.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-13T01:56:01.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-13T01:56:01.414 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-13T01:56:01.414 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2026-02-13T01:56:01.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-13T01:56:01.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:56:01.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:56:01.433 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:56:01.434 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:56:01.434 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:56:01.434 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:56:01.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2026-02-13T01:56:01.465 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-13T01:56:01.466 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-13T01:56:01.466 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T01:56:01.466 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T01:56:01.466 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-13T01:56:01.468 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-13T01:56:01.468 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T01:56:01.468 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T01:56:01.468 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T01:56:01.468 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T01:56:01.468 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:56:01.468 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:56:01.468 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T01:56:01.469 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T01:56:01.469 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get fsid 2026-02-13T01:56:01.523 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-13T01:56:01.523 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T01:56:01.523 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T01:56:01.523 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-13T01:56:01.524 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-13T01:56:01.525 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T01:56:01.525 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T01:56:01.525 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T01:56:01.525 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T01:56:01.525 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:56:01.525 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:56:01.525 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T01:56:01.525 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T01:56:01.526 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get mon_host 2026-02-13T01:56:01.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:142: TEST_allow_repair_during_recovery: run_mgr td/osd-scrub-repair x 2026-02-13T01:56:01.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-13T01:56:01.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-13T01:56:01.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-13T01:56:01.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-13T01:56:01.580 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-13T01:56:01.580 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-13T01:56:01.672 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-13T01:56:01.673 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:56:01.673 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:56:01.673 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:56:01.673 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:56:01.673 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:56:01.673 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:56:01.674 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T01:56:01.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T01:56:01.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:143: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 0 --osd_scrub_during_recovery=false --osd_debug_pretend_recovery_active=true 2026-02-13T01:56:01.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T01:56:01.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T01:56:01.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-13T01:56:01.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T01:56:01.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T01:56:01.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T01:56:01.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T01:56:01.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T01:56:01.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T01:56:01.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T01:56:01.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T01:56:01.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T01:56:01.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T01:56:01.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T01:56:01.697 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T01:56:01.697 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:56:01.698 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:56:01.698 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:56:01.698 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:56:01.698 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:56:01.698 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:56:01.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:56:01.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T01:56:01.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T01:56:01.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T01:56:01.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T01:56:01.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T01:56:01.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T01:56:01.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T01:56:01.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T01:56:01.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T01:56:01.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T01:56:01.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_debug_pretend_recovery_active=true' 2026-02-13T01:56:01.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T01:56:01.699 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T01:56:01.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a74b8d2b-3d74-4fa0-83a8-42c6bcb5fbfd 2026-02-13T01:56:01.700 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 a74b8d2b-3d74-4fa0-83a8-42c6bcb5fbfd 2026-02-13T01:56:01.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 a74b8d2b-3d74-4fa0-83a8-42c6bcb5fbfd' 2026-02-13T01:56:01.700 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T01:56:01.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCxhI5pRdoeKhAA219WrKjHLXPLQRmzeTHNJw== 2026-02-13T01:56:01.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCxhI5pRdoeKhAA219WrKjHLXPLQRmzeTHNJw=="}' 2026-02-13T01:56:01.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a74b8d2b-3d74-4fa0-83a8-42c6bcb5fbfd -i td/osd-scrub-repair/0/new.json 2026-02-13T01:56:01.803 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T01:56:01.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-13T01:56:01.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_debug_pretend_recovery_active=true --mkfs --key AQCxhI5pRdoeKhAA219WrKjHLXPLQRmzeTHNJw== --osd-uuid a74b8d2b-3d74-4fa0-83a8-42c6bcb5fbfd 2026-02-13T01:56:01.830 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:01.824+0000 7f214d4e2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:56:01.844 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:01.824+0000 7f214d4e2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:56:01.844 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:01.828+0000 7f214d4e2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:56:01.844 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:01.828+0000 7f214d4e2a40 -1 bdev(0x55e1bccf4c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-13T01:56:01.844 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:01.828+0000 7f214d4e2a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-13T01:56:02.599 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-13T01:56:02.599 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T01:56:02.600 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-13T01:56:02.600 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T01:56:02.600 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T01:56:02.701 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T01:56:02.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-13T01:56:02.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_debug_pretend_recovery_active=true 2026-02-13T01:56:02.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T01:56:02.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T01:56:02.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T01:56:02.726 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:02.720+0000 7f01d4091a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:56:02.728 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:02.720+0000 7f01d4091a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:56:02.729 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:02.724+0000 7f01d4091a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:56:02.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-13T01:56:02.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T01:56:02.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T01:56:02.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T01:56:02.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T01:56:02.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:56:02.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T01:56:02.794 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T01:56:02.795 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:56:02.795 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T01:56:02.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:56:03.118 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:03.112+0000 7f01d4091a40 -1 Falling back to public interface 2026-02-13T01:56:03.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:56:03.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:56:03.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T01:56:03.886 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T01:56:03.886 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:56:03.886 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T01:56:03.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:56:04.143 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:04.136+0000 7f01d4091a40 -1 osd.0 0 log_to_monitors true 2026-02-13T01:56:04.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:56:04.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:56:04.978 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T01:56:04.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T01:56:04.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:56:04.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T01:56:05.137 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1056929120,v1:127.0.0.1:6801/1056929120] [v2:127.0.0.1:6802/1056929120,v1:127.0.0.1:6803/1056929120] exists,up a74b8d2b-3d74-4fa0-83a8-42c6bcb5fbfd 2026-02-13T01:56:05.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T01:56:05.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T01:56:05.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T01:56:05.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:145: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 1 --osd_scrub_during_recovery=false --osd_debug_pretend_recovery_active=true 2026-02-13T01:56:05.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T01:56:05.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T01:56:05.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-13T01:56:05.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T01:56:05.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T01:56:05.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T01:56:05.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T01:56:05.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T01:56:05.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T01:56:05.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T01:56:05.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T01:56:05.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T01:56:05.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T01:56:05.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T01:56:05.142 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T01:56:05.142 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:56:05.142 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:56:05.142 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:56:05.142 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:56:05.142 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:56:05.143 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:56:05.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:56:05.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T01:56:05.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T01:56:05.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T01:56:05.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T01:56:05.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T01:56:05.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T01:56:05.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T01:56:05.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T01:56:05.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T01:56:05.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T01:56:05.144 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_debug_pretend_recovery_active=true' 2026-02-13T01:56:05.144 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T01:56:05.144 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T01:56:05.144 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3cc8a721-8931-4196-bfa7-e81c5fb4d64b 2026-02-13T01:56:05.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 3cc8a721-8931-4196-bfa7-e81c5fb4d64b' 2026-02-13T01:56:05.145 INFO:tasks.workunit.client.0.trial112.stdout:add osd1 3cc8a721-8931-4196-bfa7-e81c5fb4d64b 2026-02-13T01:56:05.145 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T01:56:05.154 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC1hI5pbCMKCRAARz1RA1rWkGK32OUlnHzIQg== 2026-02-13T01:56:05.155 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC1hI5pbCMKCRAARz1RA1rWkGK32OUlnHzIQg=="}' 2026-02-13T01:56:05.155 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3cc8a721-8931-4196-bfa7-e81c5fb4d64b -i td/osd-scrub-repair/1/new.json 2026-02-13T01:56:05.325 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T01:56:05.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-13T01:56:05.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_debug_pretend_recovery_active=true --mkfs --key AQC1hI5pbCMKCRAARz1RA1rWkGK32OUlnHzIQg== --osd-uuid 3cc8a721-8931-4196-bfa7-e81c5fb4d64b 2026-02-13T01:56:05.352 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:05.344+0000 7fcee2333a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:56:05.354 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:05.348+0000 7fcee2333a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:56:05.355 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:05.348+0000 7fcee2333a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:56:05.355 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:05.348+0000 7fcee2333a40 -1 bdev(0x5599789a8c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-13T01:56:05.356 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:05.348+0000 7fcee2333a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-13T01:56:05.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-13T01:56:05.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T01:56:05.947 INFO:tasks.workunit.client.0.trial112.stdout:adding osd1 key to auth repository 2026-02-13T01:56:05.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-13T01:56:05.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T01:56:06.155 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T01:56:06.155 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-13T01:56:06.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_debug_pretend_recovery_active=true 2026-02-13T01:56:06.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T01:56:06.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T01:56:06.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T01:56:06.180 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:06.172+0000 7fe336441a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:56:06.181 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:06.176+0000 7fe336441a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:56:06.182 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:06.176+0000 7fe336441a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:56:06.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-13T01:56:06.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T01:56:06.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T01:56:06.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T01:56:06.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T01:56:06.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:56:06.317 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T01:56:06.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T01:56:06.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:56:06.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T01:56:06.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:56:06.594 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:06.588+0000 7fe336441a40 -1 Falling back to public interface 2026-02-13T01:56:07.235 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:07.228+0000 7fe336441a40 -1 osd.1 0 log_to_monitors true 2026-02-13T01:56:07.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:56:07.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:56:07.479 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T01:56:07.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T01:56:07.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:56:07.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T01:56:07.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:56:08.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:56:08.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:56:08.641 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T01:56:08.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T01:56:08.642 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:56:08.642 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T01:56:08.802 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3984638793,v1:127.0.0.1:6811/3984638793] [v2:127.0.0.1:6812/3984638793,v1:127.0.0.1:6813/3984638793] exists,up 3cc8a721-8931-4196-bfa7-e81c5fb4d64b 2026-02-13T01:56:08.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T01:56:08.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T01:56:08.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T01:56:08.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:147: TEST_allow_repair_during_recovery: create_rbd_pool 2026-02-13T01:56:08.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-13T01:56:08.961 INFO:tasks.workunit.client.0.trial112.stderr:pool 'rbd' does not exist 2026-02-13T01:56:08.970 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-13T01:56:08.970 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-13T01:56:09.190 INFO:tasks.workunit.client.0.trial112.stderr:pool 'rbd' created 2026-02-13T01:56:09.202 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T01:56:10.203 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-13T01:56:10.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:148: TEST_allow_repair_during_recovery: wait_for_clean 2026-02-13T01:56:10.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T01:56:10.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T01:56:10.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T01:56:10.486 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T01:56:10.487 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T01:56:10.487 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T01:56:10.487 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T01:56:10.487 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T01:56:10.487 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T01:56:10.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T01:56:10.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T01:56:10.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T01:56:10.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T01:56:10.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T01:56:10.590 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T01:56:10.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T01:56:10.753 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T01:56:10.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T01:56:10.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:56:10.753 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T01:56:10.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-13T01:56:10.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-13T01:56:10.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-13T01:56:10.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:56:10.825 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T01:56:10.899 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672962 2026-02-13T01:56:10.899 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672962 2026-02-13T01:56:10.899 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-42949672962' 2026-02-13T01:56:10.899 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:56:10.900 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-13T01:56:10.900 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:56:10.901 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T01:56:10.901 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-13T01:56:10.902 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:56:10.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-13T01:56:10.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-13T01:56:10.903 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836483 2026-02-13T01:56:10.904 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:56:11.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-13T01:56:11.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:56:12.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T01:56:12.071 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:56:12.233 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-13T01:56:12.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:56:12.234 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672962 2026-02-13T01:56:12.234 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:56:12.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T01:56:12.236 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672962 2026-02-13T01:56:12.236 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:56:12.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672962 2026-02-13T01:56:12.238 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 42949672962 2026-02-13T01:56:12.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672962' 2026-02-13T01:56:12.239 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T01:56:12.397 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672962 -lt 42949672962 2026-02-13T01:56:12.398 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T01:56:12.398 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:56:12.399 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:56:12.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-13T01:56:12.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T01:56:12.594 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T01:56:12.594 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T01:56:12.594 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T01:56:12.595 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T01:56:12.595 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T01:56:12.595 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T01:56:12.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-13T01:56:12.756 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T01:56:12.757 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:56:12.757 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:56:12.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-13T01:56:12.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T01:56:12.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T01:56:12.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:150: TEST_allow_repair_during_recovery: add_something td/osd-scrub-repair rbd 2026-02-13T01:56:12.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T01:56:12.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2026-02-13T01:56:12.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2026-02-13T01:56:12.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T01:56:12.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T01:56:12.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T01:56:13.152 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T01:56:13.167 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T01:56:13.357 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T01:56:13.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T01:56:13.368 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T01:56:13.368 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-13T01:56:13.395 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:151: TEST_allow_repair_during_recovery: get_not_primary rbd SOMETHING 2026-02-13T01:56:13.395 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=rbd 2026-02-13T01:56:13.395 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-13T01:56:13.395 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary rbd SOMETHING 2026-02-13T01:56:13.395 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2026-02-13T01:56:13.395 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-13T01:56:13.396 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd SOMETHING 2026-02-13T01:56:13.396 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T01:56:13.558 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-13T01:56:13.559 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map rbd SOMETHING 2026-02-13T01:56:13.559 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-13T01:56:13.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:151: TEST_allow_repair_during_recovery: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2026-02-13T01:56:13.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:286: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2026-02-13T01:56:13.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: corrupt_and_repair_one: local poolname=rbd 2026-02-13T01:56:13.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:288: corrupt_and_repair_one: local osd=0 2026-02-13T01:56:13.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:293: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2026-02-13T01:56:13.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T01:56:13.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T01:56:13.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T01:56:13.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T01:56:13.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2026-02-13T01:56:13.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T01:56:13.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T01:56:13.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T01:56:13.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T01:56:13.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T01:56:13.727 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T01:56:13.727 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T01:56:13.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T01:56:13.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T01:56:13.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T01:56:14.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T01:56:14.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2026-02-13T01:56:14.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T01:56:14.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T01:56:14.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T01:56:14.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T01:56:14.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T01:56:14.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2026-02-13T01:56:14.208 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:eb822e21:::SOMETHING:head# 2026-02-13T01:56:14.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-13T01:56:14.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T01:56:14.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T01:56:14.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T01:56:14.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T01:56:14.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T01:56:14.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T01:56:14.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T01:56:14.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T01:56:14.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T01:56:14.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T01:56:14.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T01:56:14.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T01:56:14.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T01:56:14.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T01:56:14.305 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T01:56:14.305 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:56:14.305 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:56:14.305 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:56:14.305 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:56:14.305 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:56:14.306 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:56:14.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:56:14.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T01:56:14.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T01:56:14.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T01:56:14.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T01:56:14.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T01:56:14.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T01:56:14.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T01:56:14.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T01:56:14.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T01:56:14.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T01:56:14.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T01:56:14.308 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T01:56:14.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T01:56:14.308 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T01:56:14.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T01:56:14.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T01:56:14.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T01:56:14.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T01:56:14.324 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:14.316+0000 7f8216114a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:56:14.325 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:14.320+0000 7f8216114a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:56:14.326 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:14.320+0000 7f8216114a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:56:14.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T01:56:14.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T01:56:14.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T01:56:14.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T01:56:14.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T01:56:14.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:56:14.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T01:56:14.473 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T01:56:14.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:56:14.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T01:56:14.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:56:14.678 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:14.672+0000 7f8216114a40 -1 Falling back to public interface 2026-02-13T01:56:15.232 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:15.224+0000 7f8216114a40 -1 osd.0 19 log_to_monitors true 2026-02-13T01:56:15.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:56:15.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:56:15.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T01:56:15.634 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:56:15.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:56:15.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T01:56:15.797 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 23 up_thru 23 down_at 20 last_clean_interval [5,19) [v2:127.0.0.1:6800/480294645,v1:127.0.0.1:6801/480294645] [v2:127.0.0.1:6802/480294645,v1:127.0.0.1:6803/480294645] exists,up a74b8d2b-3d74-4fa0-83a8-42c6bcb5fbfd 2026-02-13T01:56:15.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T01:56:15.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T01:56:15.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T01:56:15.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T01:56:15.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T01:56:15.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T01:56:15.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T01:56:15.798 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T01:56:15.799 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T01:56:15.799 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T01:56:15.799 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T01:56:15.799 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T01:56:15.799 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T01:56:15.905 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T01:56:15.906 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T01:56:15.906 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T01:56:15.906 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T01:56:15.906 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T01:56:15.906 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T01:56:16.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T01:56:16.061 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T01:56:16.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T01:56:16.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:56:16.062 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T01:56:16.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247810 2026-02-13T01:56:16.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247810 2026-02-13T01:56:16.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247810' 2026-02-13T01:56:16.138 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:56:16.138 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T01:56:16.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672964 2026-02-13T01:56:16.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672964 2026-02-13T01:56:16.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247810 1-42949672964' 2026-02-13T01:56:16.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:56:16.210 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-98784247810 2026-02-13T01:56:16.210 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:56:16.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T01:56:16.212 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-98784247810 2026-02-13T01:56:16.212 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:56:16.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247810 2026-02-13T01:56:16.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 98784247810' 2026-02-13T01:56:16.213 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 98784247810 2026-02-13T01:56:16.214 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:56:16.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 98784247810 2026-02-13T01:56:16.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:56:17.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T01:56:17.379 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:56:17.537 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 98784247810 2026-02-13T01:56:17.537 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:56:18.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T01:56:18.539 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:56:18.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247810 -lt 98784247810 2026-02-13T01:56:18.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:56:18.700 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672964 2026-02-13T01:56:18.700 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:56:18.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T01:56:18.702 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672964 2026-02-13T01:56:18.702 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:56:18.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672964 2026-02-13T01:56:18.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2026-02-13T01:56:18.703 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 42949672964 2026-02-13T01:56:18.704 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T01:56:18.878 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672964 -lt 42949672964 2026-02-13T01:56:18.879 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T01:56:18.879 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:56:18.879 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:56:19.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-13T01:56:19.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T01:56:19.071 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T01:56:19.071 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T01:56:19.071 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T01:56:19.072 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T01:56:19.072 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T01:56:19.072 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T01:56:19.230 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-13T01:56:19.231 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T01:56:19.231 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:56:19.231 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:56:19.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-13T01:56:19.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T01:56:19.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T01:56:19.426 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: get_pg rbd SOMETHING 2026-02-13T01:56:19.426 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=rbd 2026-02-13T01:56:19.427 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-13T01:56:19.427 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map rbd SOMETHING 2026-02-13T01:56:19.427 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T01:56:19.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: local pg=1.3 2026-02-13T01:56:19.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: corrupt_and_repair_one: repair 1.3 2026-02-13T01:56:19.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=1.3 2026-02-13T01:56:19.592 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 1.3 2026-02-13T01:56:19.592 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-13T01:56:19.592 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T01:56:19.592 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T01:56:19.592 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-13T01:56:19.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-13T01:56:09.187011+0000 2026-02-13T01:56:19.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 1.3 2026-02-13T01:56:19.900 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 1.3 on osd.1 to repair 2026-02-13T01:56:19.910 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 1.3 2026-02-13T01:56:09.187011+0000 2026-02-13T01:56:19.910 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.3 2026-02-13T01:56:19.910 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T01:56:09.187011+0000 2026-02-13T01:56:19.910 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T01:56:19.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T01:56:19.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T01:56:19.911 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-13T01:56:19.911 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-13T01:56:19.911 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T01:56:19.911 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T01:56:19.911 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-13T01:56:20.079 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T01:56:09.187011+0000 '>' 2026-02-13T01:56:09.187011+0000 2026-02-13T01:56:20.079 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T01:56:21.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T01:56:21.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T01:56:21.081 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-13T01:56:21.081 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-13T01:56:21.081 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T01:56:21.081 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-13T01:56:21.082 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T01:56:21.242 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T01:56:09.187011+0000 '>' 2026-02-13T01:56:09.187011+0000 2026-02-13T01:56:21.242 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T01:56:22.243 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T01:56:22.243 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T01:56:22.244 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-13T01:56:22.244 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-13T01:56:22.245 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T01:56:22.245 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T01:56:22.245 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-13T01:56:22.403 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T01:56:09.187011+0000 '>' 2026-02-13T01:56:09.187011+0000 2026-02-13T01:56:22.403 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T01:56:23.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T01:56:23.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T01:56:23.405 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-13T01:56:23.405 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-13T01:56:23.405 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T01:56:23.405 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T01:56:23.405 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-13T01:56:23.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T01:56:09.187011+0000 '>' 2026-02-13T01:56:09.187011+0000 2026-02-13T01:56:23.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T01:56:24.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T01:56:24.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T01:56:24.565 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-13T01:56:24.566 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-13T01:56:24.566 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T01:56:24.566 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T01:56:24.566 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-13T01:56:24.729 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T01:56:20.348220+0000 '>' 2026-02-13T01:56:09.187011+0000 2026-02-13T01:56:24.729 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T01:56:24.729 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-13T01:56:24.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T01:56:24.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T01:56:24.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T01:56:24.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T01:56:24.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-13T01:56:24.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T01:56:24.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T01:56:24.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T01:56:24.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T01:56:24.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T01:56:24.730 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T01:56:24.730 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T01:56:24.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T01:56:24.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T01:56:24.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T01:56:25.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T01:56:25.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-13T01:56:25.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T01:56:25.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T01:56:25.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T01:56:25.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T01:56:25.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T01:56:25.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2026-02-13T01:56:25.119 INFO:tasks.workunit.client.0.trial112.stdout:_ 2026-02-13T01:56:25.119 INFO:tasks.workunit.client.0.trial112.stdout:snapset 2026-02-13T01:56:25.203 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-13T01:56:25.203 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T01:56:25.203 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T01:56:25.203 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T01:56:25.203 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T01:56:25.203 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T01:56:25.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T01:56:25.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T01:56:25.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T01:56:25.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T01:56:25.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T01:56:25.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T01:56:25.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T01:56:25.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T01:56:25.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T01:56:25.204 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T01:56:25.204 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:56:25.204 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:56:25.205 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:56:25.205 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:56:25.205 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:56:25.205 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:56:25.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:56:25.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T01:56:25.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T01:56:25.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T01:56:25.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T01:56:25.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T01:56:25.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T01:56:25.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T01:56:25.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T01:56:25.206 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T01:56:25.206 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T01:56:25.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T01:56:25.207 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T01:56:25.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T01:56:25.207 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T01:56:25.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T01:56:25.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T01:56:25.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T01:56:25.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T01:56:25.222 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:25.216+0000 7fb4f3dfca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:56:25.223 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:25.216+0000 7fb4f3dfca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:56:25.224 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:25.216+0000 7fb4f3dfca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:56:25.369 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T01:56:25.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T01:56:25.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T01:56:25.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T01:56:25.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T01:56:25.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:56:25.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T01:56:25.370 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T01:56:25.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:56:25.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T01:56:25.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:56:25.642 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:25.636+0000 7fb4f3dfca40 -1 Falling back to public interface 2026-02-13T01:56:26.322 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:26.316+0000 7fb4f3dfca40 -1 osd.1 27 log_to_monitors true 2026-02-13T01:56:26.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:56:26.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:56:26.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T01:56:26.531 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:56:26.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:56:26.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T01:56:26.695 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 30 up_thru 30 down_at 27 last_clean_interval [10,27) [v2:127.0.0.1:6810/3643103778,v1:127.0.0.1:6811/3643103778] [v2:127.0.0.1:6812/3643103778,v1:127.0.0.1:6813/3643103778] exists,up 3cc8a721-8931-4196-bfa7-e81c5fb4d64b 2026-02-13T01:56:26.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T01:56:26.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T01:56:26.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T01:56:26.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T01:56:26.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T01:56:26.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T01:56:26.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T01:56:26.696 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T01:56:26.697 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T01:56:26.697 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T01:56:26.697 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T01:56:26.697 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T01:56:26.697 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T01:56:26.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T01:56:26.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T01:56:26.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T01:56:26.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T01:56:26.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T01:56:26.792 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T01:56:26.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T01:56:26.953 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T01:56:26.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T01:56:26.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:56:26.953 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T01:56:27.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247813 2026-02-13T01:56:27.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247813 2026-02-13T01:56:27.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247813' 2026-02-13T01:56:27.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:56:27.028 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T01:56:27.099 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018882 2026-02-13T01:56:27.099 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018882 2026-02-13T01:56:27.099 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247813 1-128849018882' 2026-02-13T01:56:27.099 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:56:27.100 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-98784247813 2026-02-13T01:56:27.100 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:56:27.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T01:56:27.102 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-98784247813 2026-02-13T01:56:27.102 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:56:27.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247813 2026-02-13T01:56:27.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 98784247813' 2026-02-13T01:56:27.103 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 98784247813 2026-02-13T01:56:27.104 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:56:27.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247812 -lt 98784247813 2026-02-13T01:56:27.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:56:28.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T01:56:28.275 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:56:28.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247813 -lt 98784247813 2026-02-13T01:56:28.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:56:28.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-128849018882 2026-02-13T01:56:28.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:56:28.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T01:56:28.435 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-128849018882 2026-02-13T01:56:28.435 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:56:28.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018882 2026-02-13T01:56:28.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 128849018882' 2026-02-13T01:56:28.436 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 128849018882 2026-02-13T01:56:28.437 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T01:56:28.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018882 -lt 128849018882 2026-02-13T01:56:28.609 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T01:56:28.609 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:56:28.610 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:56:28.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-13T01:56:28.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T01:56:28.805 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T01:56:28.805 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T01:56:28.805 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T01:56:28.805 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T01:56:28.805 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T01:56:28.806 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T01:56:28.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-13T01:56:28.962 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T01:56:28.963 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:56:28.963 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:56:29.152 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-13T01:56:29.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T01:56:29.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T01:56:29.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2026-02-13T01:56:29.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-13T01:56:29.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-13T01:56:29.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T01:56:29.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T01:56:29.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T01:56:29.174 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T01:56:29.174 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T01:56:29.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T01:56:29.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T01:56:29.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T01:56:29.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T01:56:29.283 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T01:56:29.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T01:56:29.284 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T01:56:29.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T01:56:29.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T01:56:29.286 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T01:56:29.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T01:56:29.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T01:56:29.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T01:56:29.288 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T01:56:29.289 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T01:56:29.290 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T01:56:29.290 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T01:56:29.300 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T01:56:29.300 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:56:29.300 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:56:29.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T01:56:29.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T01:56:29.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T01:56:29.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-13T01:56:29.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-13T01:56:29.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-13T01:56:29.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-13T01:56:29.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T01:56:29.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T01:56:29.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T01:56:29.303 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T01:56:29.303 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T01:56:29.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T01:56:29.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T01:56:29.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T01:56:29.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T01:56:29.305 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T01:56:29.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T01:56:29.307 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T01:56:29.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T01:56:29.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T01:56:29.308 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T01:56:29.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T01:56:29.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T01:56:29.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T01:56:29.310 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T01:56:29.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T01:56:29.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T01:56:29.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T01:56:29.313 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T01:56:29.313 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:56:29.313 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:56:29.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T01:56:29.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T01:56:29.315 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T01:56:29.315 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-13T01:56:29.316 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T01:56:29.316 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:56:29.316 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:56:29.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.35207 2026-02-13T01:56:29.318 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T01:56:29.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-13T01:56:29.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T01:56:29.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-13T01:56:29.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_basic td/osd-scrub-repair 2026-02-13T01:56:29.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:498: TEST_auto_repair_bluestore_basic: local dir=td/osd-scrub-repair 2026-02-13T01:56:29.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:499: TEST_auto_repair_bluestore_basic: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='1' ['pool_name']='testpool' ['extras']=' --osd_scrub_auto_repair=true') 2026-02-13T01:56:29.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:499: TEST_auto_repair_bluestore_basic: local -A cluster_conf 2026-02-13T01:56:29.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:505: TEST_auto_repair_bluestore_basic: standard_scrub_cluster td/osd-scrub-repair cluster_conf 2026-02-13T01:56:29.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local dir=td/osd-scrub-repair 2026-02-13T01:56:29.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:230: standard_scrub_cluster: local -n args=cluster_conf 2026-02-13T01:56:29.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local OSDS=3 2026-02-13T01:56:29.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: local pg_num=1 2026-02-13T01:56:29.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:235: standard_scrub_cluster: local poolname=testpool 2026-02-13T01:56:29.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:236: standard_scrub_cluster: args['pool_name']=testpool 2026-02-13T01:56:29.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:238: standard_scrub_cluster: local pool_default_size=3 2026-02-13T01:56:29.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:239: standard_scrub_cluster: args['pool_default_size']=3 2026-02-13T01:56:29.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:241: standard_scrub_cluster: local 'extra_pars= --osd_scrub_auto_repair=true' 2026-02-13T01:56:29.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:242: standard_scrub_cluster: local debug_msg=dbg 2026-02-13T01:56:29.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:245: standard_scrub_cluster: local saved_echo_flag=x 2026-02-13T01:56:29.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:246: standard_scrub_cluster: set +x 2026-02-13T01:56:29.598 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 b0794629-8023-47c2-a2b2-e5a4a450d316 2026-02-13T01:56:29.700 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T01:56:29.730 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:29.724+0000 7fa592b87a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:56:29.731 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:29.724+0000 7fa592b87a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:56:29.732 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:29.724+0000 7fa592b87a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:56:29.733 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:29.724+0000 7fa592b87a40 -1 bdev(0x55e562a54c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-13T01:56:29.733 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:29.728+0000 7fa592b87a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-13T01:56:30.474 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T01:56:30.580 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T01:56:30.612 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:30.604+0000 7f36e83cda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:56:30.614 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:30.608+0000 7f36e83cda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:56:30.616 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:30.608+0000 7f36e83cda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:56:30.676 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T01:56:31.158 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:31.152+0000 7f36e83cda40 -1 Falling back to public interface 2026-02-13T01:56:31.765 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T01:56:32.100 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:32.092+0000 7f36e83cda40 -1 osd.0 0 log_to_monitors true 2026-02-13T01:56:32.856 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T01:56:33.013 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2737284504,v1:127.0.0.1:6801/2737284504] [v2:127.0.0.1:6802/2737284504,v1:127.0.0.1:6803/2737284504] exists,up b0794629-8023-47c2-a2b2-e5a4a450d316 2026-02-13T01:56:33.019 INFO:tasks.workunit.client.0.trial112.stdout:add osd1 56bb89c0-22fb-4a82-89f0-a2c3fe4a4e55 2026-02-13T01:56:33.199 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T01:56:33.237 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:33.232+0000 7fae6b11da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:56:33.239 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:33.232+0000 7fae6b11da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:56:33.240 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:33.232+0000 7fae6b11da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:56:33.240 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:33.236+0000 7fae6b11da40 -1 bdev(0x5569cb5fec00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-13T01:56:33.241 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:33.236+0000 7fae6b11da40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-13T01:56:33.880 INFO:tasks.workunit.client.0.trial112.stdout:adding osd1 key to auth repository 2026-02-13T01:56:34.085 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T01:56:34.115 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:34.108+0000 7f859a3c7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:56:34.117 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:34.112+0000 7f859a3c7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:56:34.118 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:34.112+0000 7f859a3c7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:56:34.248 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T01:56:34.458 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:34.452+0000 7f859a3c7a40 -1 Falling back to public interface 2026-02-13T01:56:35.049 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:35.044+0000 7f859a3c7a40 -1 osd.1 0 log_to_monitors true 2026-02-13T01:56:35.412 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T01:56:36.577 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T01:56:36.736 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3875445605,v1:127.0.0.1:6811/3875445605] [v2:127.0.0.1:6812/3875445605,v1:127.0.0.1:6813/3875445605] exists,up 56bb89c0-22fb-4a82-89f0-a2c3fe4a4e55 2026-02-13T01:56:36.742 INFO:tasks.workunit.client.0.trial112.stdout:add osd2 da5e8d14-12fe-473f-b4ad-27fab78e8869 2026-02-13T01:56:36.907 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T01:56:36.937 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:36.932+0000 7f355b8cba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:56:36.939 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:36.932+0000 7f355b8cba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:56:36.940 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:36.932+0000 7f355b8cba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:56:36.940 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:36.936+0000 7f355b8cba40 -1 bdev(0x55a6719fcc00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-13T01:56:36.940 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:36.936+0000 7f355b8cba40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-13T01:56:37.552 INFO:tasks.workunit.client.0.trial112.stdout:adding osd2 key to auth repository 2026-02-13T01:56:37.752 INFO:tasks.workunit.client.0.trial112.stdout:start osd.2 2026-02-13T01:56:37.783 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:37.776+0000 7ff2a7357a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:56:37.785 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:37.780+0000 7ff2a7357a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:56:37.786 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:37.780+0000 7ff2a7357a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:56:37.911 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T01:56:38.178 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:38.172+0000 7ff2a7357a40 -1 Falling back to public interface 2026-02-13T01:56:38.803 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:38.796+0000 7ff2a7357a40 -1 osd.2 0 log_to_monitors true 2026-02-13T01:56:39.074 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T01:56:40.237 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T01:56:40.393 INFO:tasks.workunit.client.0.trial112.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2660055268,v1:127.0.0.1:6819/2660055268] [v2:127.0.0.1:6820/2660055268,v1:127.0.0.1:6821/2660055268] exists,up da5e8d14-12fe-473f-b4ad-27fab78e8869 2026-02-13T01:56:40.603 INFO:tasks.workunit.client.0.trial112.stderr:pool 'testpool' created 2026-02-13T01:56:42.104 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836490 2026-02-13T01:56:44.604 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705671 2026-02-13T01:56:44.774 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542148 2026-02-13T01:56:46.667 INFO:tasks.workunit.client.0.trial112.stdout:standard_scrub_cluster: dbg: test pool is testpool 1 2026-02-13T01:56:46.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:506: TEST_auto_repair_bluestore_basic: local poolid=1 2026-02-13T01:56:46.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:507: TEST_auto_repair_bluestore_basic: local poolname=testpool 2026-02-13T01:56:46.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:509: TEST_auto_repair_bluestore_basic: ceph osd pool set testpool size 2 2026-02-13T01:56:46.847 INFO:tasks.workunit.client.0.trial112.stderr:set pool 1 size to 2 2026-02-13T01:56:46.857 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:510: TEST_auto_repair_bluestore_basic: wait_for_clean 2026-02-13T01:56:46.857 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T01:56:46.857 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T01:56:46.857 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T01:56:46.857 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T01:56:46.858 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T01:56:46.858 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T01:56:46.858 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T01:56:46.858 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T01:56:46.859 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T01:56:46.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T01:56:46.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T01:56:46.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T01:56:46.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T01:56:46.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T01:56:46.975 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T01:56:47.138 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T01:56:47.139 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:56:47.139 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T01:56:47.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T01:56:47.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:56:47.139 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T01:56:47.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836496 2026-02-13T01:56:47.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836496 2026-02-13T01:56:47.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836496' 2026-02-13T01:56:47.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:56:47.215 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T01:56:47.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705678 2026-02-13T01:56:47.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705678 2026-02-13T01:56:47.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836496 1-38654705678' 2026-02-13T01:56:47.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:56:47.287 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T01:56:47.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542154 2026-02-13T01:56:47.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542154 2026-02-13T01:56:47.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836496 1-38654705678 2-60129542154' 2026-02-13T01:56:47.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:56:47.361 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836496 2026-02-13T01:56:47.361 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:56:47.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T01:56:47.363 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836496 2026-02-13T01:56:47.363 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:56:47.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836496 2026-02-13T01:56:47.364 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836496 2026-02-13T01:56:47.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2026-02-13T01:56:47.365 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:56:47.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836494 -lt 21474836496 2026-02-13T01:56:47.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:56:48.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T01:56:48.523 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:56:48.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836497 -lt 21474836496 2026-02-13T01:56:48.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:56:48.689 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705678 2026-02-13T01:56:48.689 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:56:48.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T01:56:48.690 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705678 2026-02-13T01:56:48.690 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:56:48.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705678 2026-02-13T01:56:48.692 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705678 2026-02-13T01:56:48.692 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705678' 2026-02-13T01:56:48.692 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T01:56:48.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705678 -lt 38654705678 2026-02-13T01:56:48.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:56:48.864 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542154 2026-02-13T01:56:48.865 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:56:48.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T01:56:48.866 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542154 2026-02-13T01:56:48.866 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:56:48.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542154 2026-02-13T01:56:48.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542154' 2026-02-13T01:56:48.868 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542154 2026-02-13T01:56:48.869 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T01:56:49.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542154 -lt 60129542154 2026-02-13T01:56:49.035 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T01:56:49.035 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:56:49.036 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:56:49.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T01:56:49.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T01:56:49.234 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T01:56:49.234 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T01:56:49.234 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T01:56:49.234 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T01:56:49.235 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T01:56:49.235 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T01:56:49.395 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T01:56:49.395 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T01:56:49.396 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:56:49.396 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:56:49.594 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T01:56:49.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T01:56:49.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T01:56:49.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:513: TEST_auto_repair_bluestore_basic: local payload=ABCDEF 2026-02-13T01:56:49.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:514: TEST_auto_repair_bluestore_basic: echo ABCDEF 2026-02-13T01:56:49.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:515: TEST_auto_repair_bluestore_basic: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-13T01:56:49.614 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:519: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2026-02-13T01:56:49.615 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-13T01:56:49.615 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-13T01:56:49.615 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2026-02-13T01:56:49.615 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-13T01:56:49.615 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-13T01:56:49.616 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-13T01:56:49.616 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T01:56:49.775 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-13T01:56:49.776 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2026-02-13T01:56:49.776 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-13T01:56:49.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:519: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2026-02-13T01:56:49.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T01:56:49.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T01:56:49.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T01:56:49.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T01:56:49.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2026-02-13T01:56:49.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T01:56:49.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T01:56:49.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T01:56:49.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T01:56:49.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T01:56:49.939 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T01:56:49.939 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T01:56:49.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T01:56:49.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T01:56:49.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T01:56:50.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T01:56:50.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2026-02-13T01:56:50.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T01:56:50.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T01:56:50.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T01:56:50.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T01:56:50.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T01:56:50.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2026-02-13T01:56:50.444 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:eb822e21:::SOMETHING:head# 2026-02-13T01:56:50.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-13T01:56:50.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T01:56:50.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T01:56:50.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T01:56:50.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T01:56:50.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T01:56:50.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T01:56:50.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T01:56:50.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T01:56:50.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T01:56:50.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T01:56:50.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T01:56:50.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T01:56:50.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T01:56:50.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T01:56:50.535 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T01:56:50.535 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:56:50.535 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:56:50.535 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:56:50.535 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:56:50.536 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:56:50.536 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:56:50.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:56:50.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T01:56:50.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T01:56:50.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T01:56:50.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T01:56:50.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T01:56:50.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T01:56:50.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T01:56:50.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T01:56:50.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T01:56:50.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T01:56:50.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T01:56:50.539 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T01:56:50.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T01:56:50.539 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T01:56:50.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T01:56:50.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T01:56:50.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T01:56:50.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T01:56:50.553 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:50.548+0000 7f0f0543ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:56:50.555 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:50.548+0000 7f0f0543ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:56:50.556 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:50.552+0000 7f0f0543ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:56:50.694 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T01:56:50.694 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T01:56:50.694 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T01:56:50.694 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T01:56:50.694 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T01:56:50.694 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:56:50.694 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T01:56:50.695 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T01:56:50.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:56:50.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T01:56:50.859 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:56:50.998 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:50.992+0000 7f0f0543ca40 -1 Falling back to public interface 2026-02-13T01:56:51.549 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:56:51.544+0000 7f0f0543ca40 -1 osd.0 21 log_to_monitors true 2026-02-13T01:56:51.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:56:51.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:56:51.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T01:56:51.861 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:56:51.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:56:51.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T01:56:52.021 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 21 last_clean_interval [5,21) [v2:127.0.0.1:6800/3912471445,v1:127.0.0.1:6801/3912471445] [v2:127.0.0.1:6802/3912471445,v1:127.0.0.1:6803/3912471445] exists,up b0794629-8023-47c2-a2b2-e5a4a450d316 2026-02-13T01:56:52.021 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T01:56:52.021 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T01:56:52.021 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T01:56:52.021 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T01:56:52.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T01:56:52.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T01:56:52.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T01:56:52.022 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T01:56:52.023 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T01:56:52.023 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T01:56:52.023 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T01:56:52.023 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T01:56:52.023 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T01:56:52.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T01:56:52.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T01:56:52.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T01:56:52.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T01:56:52.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T01:56:52.134 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T01:56:52.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T01:56:52.302 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:56:52.302 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T01:56:52.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T01:56:52.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:56:52.303 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T01:56:52.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215106 2026-02-13T01:56:52.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215106 2026-02-13T01:56:52.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215106' 2026-02-13T01:56:52.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:56:52.375 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T01:56:52.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705684 2026-02-13T01:56:52.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705684 2026-02-13T01:56:52.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215106 1-38654705684' 2026-02-13T01:56:52.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:56:52.451 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T01:56:52.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542160 2026-02-13T01:56:52.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542160 2026-02-13T01:56:52.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215106 1-38654705684 2-60129542160' 2026-02-13T01:56:52.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:56:52.535 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-103079215106 2026-02-13T01:56:52.535 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:56:52.536 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T01:56:52.537 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-103079215106 2026-02-13T01:56:52.537 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:56:52.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215106 2026-02-13T01:56:52.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2026-02-13T01:56:52.539 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 103079215106 2026-02-13T01:56:52.539 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:56:52.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 103079215106 2026-02-13T01:56:52.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:56:53.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T01:56:53.709 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:56:53.876 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215108 -lt 103079215106 2026-02-13T01:56:53.876 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:56:53.877 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705684 2026-02-13T01:56:53.877 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:56:53.878 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T01:56:53.879 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705684 2026-02-13T01:56:53.879 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:56:53.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705684 2026-02-13T01:56:53.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705684' 2026-02-13T01:56:53.881 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 38654705684 2026-02-13T01:56:53.881 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T01:56:54.042 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705685 -lt 38654705684 2026-02-13T01:56:54.042 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:56:54.042 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542160 2026-02-13T01:56:54.043 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:56:54.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T01:56:54.044 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542160 2026-02-13T01:56:54.045 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:56:54.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542160 2026-02-13T01:56:54.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542160' 2026-02-13T01:56:54.046 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 60129542160 2026-02-13T01:56:54.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T01:56:54.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542161 -lt 60129542160 2026-02-13T01:56:54.215 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T01:56:54.215 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:56:54.215 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:56:54.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T01:56:54.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T01:56:54.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T01:56:54.408 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T01:56:54.408 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T01:56:54.408 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T01:56:54.408 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T01:56:54.408 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T01:56:54.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T01:56:54.562 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T01:56:54.562 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:56:54.562 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:56:54.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T01:56:54.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T01:56:54.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T01:56:54.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:520: TEST_auto_repair_bluestore_basic: ceph tell 'osd.*' config set osd_scrub_auto_repair true 2026-02-13T01:56:54.823 INFO:tasks.workunit.client.0.trial112.stdout:osd.0: { 2026-02-13T01:56:54.823 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-13T01:56:54.823 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T01:56:54.831 INFO:tasks.workunit.client.0.trial112.stdout:osd.1: { 2026-02-13T01:56:54.831 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2026-02-13T01:56:54.831 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T01:56:54.838 INFO:tasks.workunit.client.0.trial112.stdout:osd.2: { 2026-02-13T01:56:54.838 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2026-02-13T01:56:54.838 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T01:56:54.846 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:522: TEST_auto_repair_bluestore_basic: get_pg testpool SOMETHING 2026-02-13T01:56:54.847 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2026-02-13T01:56:54.847 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-13T01:56:54.847 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool SOMETHING 2026-02-13T01:56:54.847 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T01:56:55.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:522: TEST_auto_repair_bluestore_basic: local pgid=1.0 2026-02-13T01:56:55.008 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:523: TEST_auto_repair_bluestore_basic: get_primary testpool SOMETHING 2026-02-13T01:56:55.008 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-13T01:56:55.008 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-13T01:56:55.008 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-13T01:56:55.008 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T01:56:55.167 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:523: TEST_auto_repair_bluestore_basic: local primary=1 2026-02-13T01:56:55.168 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:524: TEST_auto_repair_bluestore_basic: get_last_scrub_stamp 1.0 2026-02-13T01:56:55.168 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T01:56:55.168 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T01:56:55.168 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T01:56:55.168 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T01:56:55.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:524: TEST_auto_repair_bluestore_basic: local last_scrub_stamp=2026-02-13T01:56:40.600780+0000 2026-02-13T01:56:55.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:527: TEST_auto_repair_bluestore_basic: ceph tell 1.0 schedule-deep-scrub 2026-02-13T01:56:55.390 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T01:56:55.390 INFO:tasks.workunit.client.0.trial112.stdout: "deep": true, 2026-02-13T01:56:55.390 INFO:tasks.workunit.client.0.trial112.stdout: "must": false, 2026-02-13T01:56:55.390 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "2026-02-01T21:08:45.389037+0000" 2026-02-13T01:56:55.390 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T01:56:55.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:530: TEST_auto_repair_bluestore_basic: wait_for_scrub 1.0 2026-02-13T01:56:40.600780+0000 2026-02-13T01:56:55.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-13T01:56:55.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T01:56:40.600780+0000 2026-02-13T01:56:55.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T01:56:55.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T01:56:55.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T01:56:55.399 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T01:56:55.399 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T01:56:55.399 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T01:56:55.399 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T01:56:55.400 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T01:56:55.556 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T01:56:40.600780+0000 '>' 2026-02-13T01:56:40.600780+0000 2026-02-13T01:56:55.557 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T01:56:56.558 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T01:56:56.558 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T01:56:56.559 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T01:56:56.559 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T01:56:56.559 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T01:56:56.559 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T01:56:56.559 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T01:56:56.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T01:56:40.600780+0000 '>' 2026-02-13T01:56:40.600780+0000 2026-02-13T01:56:56.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T01:56:57.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T01:56:57.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T01:56:57.718 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T01:56:57.718 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T01:56:57.719 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T01:56:57.719 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T01:56:57.719 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T01:56:57.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T01:56:56.043374+0000 '>' 2026-02-13T01:56:40.600780+0000 2026-02-13T01:56:57.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T01:56:57.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:531: TEST_auto_repair_bluestore_basic: wait_for_clean 2026-02-13T01:56:57.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T01:56:57.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T01:56:57.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T01:56:57.883 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T01:56:57.884 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T01:56:57.884 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T01:56:57.884 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T01:56:57.884 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T01:56:57.884 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T01:56:57.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T01:56:57.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T01:56:57.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T01:56:57.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T01:56:57.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T01:56:57.985 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T01:56:58.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T01:56:58.149 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:56:58.149 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T01:56:58.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T01:56:58.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:56:58.149 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T01:56:58.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215113 2026-02-13T01:56:58.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215113 2026-02-13T01:56:58.222 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215113' 2026-02-13T01:56:58.222 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:56:58.222 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T01:56:58.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705691 2026-02-13T01:56:58.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705691 2026-02-13T01:56:58.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215113 1-38654705691' 2026-02-13T01:56:58.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:56:58.299 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T01:56:58.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542167 2026-02-13T01:56:58.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542167 2026-02-13T01:56:58.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215113 1-38654705691 2-60129542167' 2026-02-13T01:56:58.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:56:58.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-103079215113 2026-02-13T01:56:58.375 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:56:58.376 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T01:56:58.376 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-103079215113 2026-02-13T01:56:58.376 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:56:58.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215113 2026-02-13T01:56:58.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 103079215113' 2026-02-13T01:56:58.378 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 103079215113 2026-02-13T01:56:58.378 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:56:58.536 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215112 -lt 103079215113 2026-02-13T01:56:58.536 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:56:59.537 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T01:56:59.538 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:56:59.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215112 -lt 103079215113 2026-02-13T01:56:59.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:57:00.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T01:57:00.702 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:57:00.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215115 -lt 103079215113 2026-02-13T01:57:00.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:57:00.866 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705691 2026-02-13T01:57:00.866 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:57:00.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T01:57:00.868 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705691 2026-02-13T01:57:00.868 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:57:00.869 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705691 2026-02-13T01:57:00.869 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705691 2026-02-13T01:57:00.869 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705691' 2026-02-13T01:57:00.869 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T01:57:01.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705692 -lt 38654705691 2026-02-13T01:57:01.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:57:01.031 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542167 2026-02-13T01:57:01.031 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:57:01.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T01:57:01.033 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542167 2026-02-13T01:57:01.033 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:57:01.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542167 2026-02-13T01:57:01.035 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542167 2026-02-13T01:57:01.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542167' 2026-02-13T01:57:01.035 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T01:57:01.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542168 -lt 60129542167 2026-02-13T01:57:01.209 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T01:57:01.210 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:57:01.210 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:57:01.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T01:57:01.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T01:57:01.412 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T01:57:01.412 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T01:57:01.412 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T01:57:01.412 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T01:57:01.412 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T01:57:01.413 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T01:57:01.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T01:57:01.570 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T01:57:01.571 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:57:01.571 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:57:01.771 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T01:57:01.771 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T01:57:01.771 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T01:57:01.771 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:532: TEST_auto_repair_bluestore_basic: ceph pg dump pgs 2026-02-13T01:57:01.922 INFO:tasks.workunit.client.0.trial112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-13T01:57:01.922 INFO:tasks.workunit.client.0.trial112.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2026-02-13T01:56:56.045739+0000 20'1 26:71 [1,0] 1 [1,0] 1 20'1 2026-02-13T01:56:56.043374+0000 20'1 2026-02-13T01:56:56.043374+0000 0 1 periodic scrub scheduled @ 2026-02-14T01:56:56.043374+0000 1 0 2026-02-13T01:57:01.923 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:57:01.923 INFO:tasks.workunit.client.0.trial112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-13T01:57:01.923 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T01:57:01.935 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:535: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2026-02-13T01:57:01.935 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-13T01:57:01.935 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-13T01:57:01.936 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2026-02-13T01:57:01.936 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-13T01:57:01.936 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-13T01:57:01.936 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-13T01:57:01.936 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T01:57:02.095 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-13T01:57:02.095 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2026-02-13T01:57:02.096 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-13T01:57:02.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:535: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-13T01:57:02.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T01:57:02.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T01:57:02.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T01:57:02.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T01:57:02.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-13T01:57:02.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T01:57:02.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T01:57:02.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T01:57:02.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T01:57:02.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T01:57:02.264 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T01:57:02.264 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T01:57:02.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T01:57:02.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T01:57:02.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T01:57:02.572 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T01:57:02.572 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-13T01:57:02.572 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T01:57:02.572 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T01:57:02.572 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T01:57:02.572 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T01:57:02.572 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T01:57:02.572 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2026-02-13T01:57:02.643 INFO:tasks.workunit.client.0.trial112.stdout:_ 2026-02-13T01:57:02.643 INFO:tasks.workunit.client.0.trial112.stdout:snapset 2026-02-13T01:57:02.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-13T01:57:02.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T01:57:02.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T01:57:02.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T01:57:02.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T01:57:02.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T01:57:02.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T01:57:02.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T01:57:02.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T01:57:02.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T01:57:02.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T01:57:02.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T01:57:02.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T01:57:02.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T01:57:02.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T01:57:02.722 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T01:57:02.722 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:57:02.722 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:57:02.722 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:57:02.723 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:57:02.723 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:57:02.724 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:57:02.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:57:02.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T01:57:02.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T01:57:02.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T01:57:02.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T01:57:02.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T01:57:02.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T01:57:02.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T01:57:02.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T01:57:02.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T01:57:02.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T01:57:02.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T01:57:02.725 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T01:57:02.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T01:57:02.726 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T01:57:02.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T01:57:02.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T01:57:02.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T01:57:02.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T01:57:02.745 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:02.740+0000 7fcd61b24a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:57:02.746 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:02.740+0000 7fcd61b24a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:57:02.747 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:02.740+0000 7fcd61b24a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:57:02.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T01:57:02.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T01:57:02.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T01:57:02.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T01:57:02.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T01:57:02.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:57:02.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T01:57:02.891 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T01:57:02.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:57:02.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T01:57:03.050 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:57:03.170 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:03.164+0000 7fcd61b24a40 -1 Falling back to public interface 2026-02-13T01:57:03.742 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:03.736+0000 7fcd61b24a40 -1 osd.0 26 log_to_monitors true 2026-02-13T01:57:04.052 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:57:04.052 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:57:04.052 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T01:57:04.052 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:57:04.052 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:57:04.052 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T01:57:04.214 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [24,26) [v2:127.0.0.1:6800/3204549967,v1:127.0.0.1:6801/3204549967] [v2:127.0.0.1:6802/3204549967,v1:127.0.0.1:6803/3204549967] exists,up b0794629-8023-47c2-a2b2-e5a4a450d316 2026-02-13T01:57:04.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T01:57:04.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T01:57:04.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T01:57:04.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T01:57:04.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T01:57:04.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T01:57:04.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T01:57:04.215 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T01:57:04.216 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T01:57:04.216 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T01:57:04.216 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T01:57:04.216 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T01:57:04.216 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T01:57:04.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T01:57:04.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T01:57:04.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T01:57:04.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T01:57:04.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T01:57:04.326 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T01:57:04.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T01:57:04.489 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:57:04.489 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T01:57:04.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T01:57:04.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:57:04.489 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T01:57:04.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018882 2026-02-13T01:57:04.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018882 2026-02-13T01:57:04.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018882' 2026-02-13T01:57:04.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:57:04.566 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T01:57:04.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705698 2026-02-13T01:57:04.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705698 2026-02-13T01:57:04.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018882 1-38654705698' 2026-02-13T01:57:04.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:57:04.638 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T01:57:04.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542174 2026-02-13T01:57:04.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542174 2026-02-13T01:57:04.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018882 1-38654705698 2-60129542174' 2026-02-13T01:57:04.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:57:04.712 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-128849018882 2026-02-13T01:57:04.712 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:57:04.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T01:57:04.713 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-128849018882 2026-02-13T01:57:04.714 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:57:04.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018882 2026-02-13T01:57:04.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2026-02-13T01:57:04.715 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 128849018882 2026-02-13T01:57:04.716 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:57:04.877 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215117 -lt 128849018882 2026-02-13T01:57:04.877 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:57:05.878 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T01:57:05.878 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:57:06.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018883 -lt 128849018882 2026-02-13T01:57:06.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:57:06.044 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705698 2026-02-13T01:57:06.045 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:57:06.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T01:57:06.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705698 2026-02-13T01:57:06.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:57:06.048 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705698 2026-02-13T01:57:06.048 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705698' 2026-02-13T01:57:06.048 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 38654705698 2026-02-13T01:57:06.049 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T01:57:06.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705699 -lt 38654705698 2026-02-13T01:57:06.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:57:06.210 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542174 2026-02-13T01:57:06.210 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:57:06.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T01:57:06.212 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542174 2026-02-13T01:57:06.212 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:57:06.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542174 2026-02-13T01:57:06.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542174' 2026-02-13T01:57:06.214 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 60129542174 2026-02-13T01:57:06.214 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T01:57:06.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542175 -lt 60129542174 2026-02-13T01:57:06.388 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T01:57:06.389 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:57:06.389 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:57:06.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T01:57:06.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T01:57:06.590 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T01:57:06.590 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T01:57:06.591 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T01:57:06.591 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T01:57:06.591 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T01:57:06.591 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T01:57:06.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T01:57:06.750 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T01:57:06.750 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:57:06.750 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:57:06.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T01:57:06.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T01:57:06.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T01:57:06.952 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:536: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2026-02-13T01:57:06.952 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-13T01:57:06.952 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-13T01:57:06.953 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2026-02-13T01:57:06.953 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-13T01:57:06.953 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-13T01:57:06.953 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-13T01:57:06.953 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T01:57:07.113 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-13T01:57:07.114 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2026-02-13T01:57:07.114 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-13T01:57:07.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:536: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2026-02-13T01:57:07.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T01:57:07.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T01:57:07.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T01:57:07.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T01:57:07.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2026-02-13T01:57:07.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T01:57:07.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T01:57:07.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T01:57:07.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T01:57:07.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T01:57:07.282 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T01:57:07.282 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T01:57:07.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T01:57:07.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T01:57:07.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T01:57:07.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T01:57:07.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2026-02-13T01:57:07.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T01:57:07.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T01:57:07.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T01:57:07.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T01:57:07.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T01:57:07.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2026-02-13T01:57:07.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-13T01:57:07.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T01:57:07.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T01:57:07.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T01:57:07.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T01:57:07.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T01:57:07.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T01:57:07.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T01:57:07.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T01:57:07.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T01:57:07.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T01:57:07.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T01:57:07.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T01:57:07.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T01:57:07.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T01:57:07.787 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T01:57:07.788 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:57:07.788 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:57:07.788 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:57:07.788 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:57:07.788 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:57:07.788 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:57:07.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:57:07.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T01:57:07.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T01:57:07.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T01:57:07.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T01:57:07.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T01:57:07.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T01:57:07.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T01:57:07.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T01:57:07.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T01:57:07.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T01:57:07.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T01:57:07.790 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T01:57:07.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T01:57:07.790 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T01:57:07.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T01:57:07.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T01:57:07.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T01:57:07.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T01:57:07.811 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:07.804+0000 7f4e23b8ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:57:07.813 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:07.808+0000 7f4e23b8ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:57:07.814 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:07.808+0000 7f4e23b8ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:57:07.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T01:57:07.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T01:57:07.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T01:57:07.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T01:57:07.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T01:57:07.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:57:07.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T01:57:07.953 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T01:57:07.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:57:07.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T01:57:08.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:57:08.225 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:08.220+0000 7f4e23b8ba40 -1 Falling back to public interface 2026-02-13T01:57:08.856 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:08.852+0000 7f4e23b8ba40 -1 osd.0 31 log_to_monitors true 2026-02-13T01:57:09.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:57:09.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:57:09.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T01:57:09.112 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:57:09.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:57:09.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T01:57:09.267 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 35 up_thru 0 down_at 32 last_clean_interval [30,31) [v2:127.0.0.1:6800/1207209428,v1:127.0.0.1:6801/1207209428] [v2:127.0.0.1:6802/1207209428,v1:127.0.0.1:6803/1207209428] exists,up b0794629-8023-47c2-a2b2-e5a4a450d316 2026-02-13T01:57:09.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T01:57:09.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T01:57:09.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T01:57:09.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T01:57:09.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T01:57:09.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T01:57:09.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T01:57:09.268 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T01:57:09.269 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T01:57:09.269 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T01:57:09.269 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T01:57:09.269 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T01:57:09.270 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T01:57:09.369 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T01:57:09.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T01:57:09.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T01:57:09.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T01:57:09.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T01:57:09.370 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T01:57:09.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T01:57:09.531 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:57:09.532 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T01:57:09.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T01:57:09.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:57:09.532 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T01:57:09.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855362 2026-02-13T01:57:09.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855362 2026-02-13T01:57:09.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-150323855362' 2026-02-13T01:57:09.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:57:09.605 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T01:57:09.680 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705704 2026-02-13T01:57:09.680 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705704 2026-02-13T01:57:09.680 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-150323855362 1-38654705704' 2026-02-13T01:57:09.680 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:57:09.680 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T01:57:09.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542180 2026-02-13T01:57:09.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542180 2026-02-13T01:57:09.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-150323855362 1-38654705704 2-60129542180' 2026-02-13T01:57:09.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:57:09.752 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-150323855362 2026-02-13T01:57:09.752 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:57:09.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T01:57:09.754 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-150323855362 2026-02-13T01:57:09.754 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:57:09.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855362 2026-02-13T01:57:09.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2026-02-13T01:57:09.756 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 150323855362 2026-02-13T01:57:09.756 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:57:09.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855362 -lt 150323855362 2026-02-13T01:57:09.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:57:09.921 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705704 2026-02-13T01:57:09.921 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:57:09.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T01:57:09.923 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705704 2026-02-13T01:57:09.923 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:57:09.925 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705704 2026-02-13T01:57:09.925 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705704' 2026-02-13T01:57:09.925 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 38654705704 2026-02-13T01:57:09.925 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T01:57:10.086 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705704 -lt 38654705704 2026-02-13T01:57:10.086 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:57:10.087 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542180 2026-02-13T01:57:10.087 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:57:10.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T01:57:10.089 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542180 2026-02-13T01:57:10.089 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:57:10.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542180 2026-02-13T01:57:10.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542180' 2026-02-13T01:57:10.090 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 60129542180 2026-02-13T01:57:10.091 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T01:57:10.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542180 -lt 60129542180 2026-02-13T01:57:10.251 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T01:57:10.251 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:57:10.251 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:57:10.439 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T01:57:10.439 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T01:57:10.440 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T01:57:10.440 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T01:57:10.440 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T01:57:10.440 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T01:57:10.440 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T01:57:10.440 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T01:57:10.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-13T01:57:10.596 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T01:57:10.596 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:57:10.596 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:57:10.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-13T01:57:10.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' -1 2026-02-13T01:57:10.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-13T01:57:10.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=0 2026-02-13T01:57:10.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-13T01:57:10.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-13T01:57:10.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-13T01:57:10.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T01:57:10.892 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T01:57:10.892 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T01:57:10.892 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T01:57:10.893 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T01:57:10.893 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T01:57:10.893 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T01:57:11.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-13T01:57:11.046 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T01:57:11.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:57:11.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:57:11.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-13T01:57:11.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-13T01:57:11.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-13T01:57:11.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-13T01:57:11.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-13T01:57:11.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-13T01:57:11.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-13T01:57:11.249 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-13T01:57:11.249 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-13T01:57:11.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-13T01:57:11.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-13T01:57:11.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-13T01:57:11.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-13T01:57:11.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-13T01:57:11.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-13T01:57:11.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T01:57:11.652 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T01:57:11.652 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T01:57:11.652 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T01:57:11.652 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T01:57:11.652 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T01:57:11.652 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T01:57:11.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-13T01:57:11.814 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T01:57:11.814 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:57:11.814 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:57:12.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-13T01:57:12.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-13T01:57:12.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-13T01:57:12.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-13T01:57:12.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-13T01:57:12.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-13T01:57:12.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-13T01:57:12.010 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-13T01:57:12.010 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-13T01:57:12.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-13T01:57:12.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-13T01:57:12.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 2 >= 13 )) 2026-02-13T01:57:12.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-13T01:57:12.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.4 2026-02-13T01:57:12.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-13T01:57:12.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T01:57:12.607 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T01:57:12.607 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T01:57:12.607 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T01:57:12.607 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T01:57:12.607 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T01:57:12.607 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T01:57:12.768 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T01:57:12.769 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T01:57:12.769 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:57:12.770 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:57:12.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T01:57:12.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T01:57:12.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T01:57:12.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:537: TEST_auto_repair_bluestore_basic: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-13T01:57:12.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:538: TEST_auto_repair_bluestore_basic: grep scrub_finish td/osd-scrub-repair/osd.1.log 2026-02-13T01:57:12.965 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:56:56.040+0000 7f857bb4c640 10 osd.1 pg_epoch: 26 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep(periodic-regular,auto-repair)] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2026-02-13T01:57:12.966 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:56:56.040+0000 7f857bb4c640 10 osd.1 pg_epoch: 26 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep(periodic-regular,auto-repair)] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2026-02-13T01:57:12.966 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:56:56.040+0000 7f857bb4c640 15 osd.1 pg_epoch: 26 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2026-02-13T01:57:12.966 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:56:56.040+0000 7f857bb4c640 20 osd.1 pg_epoch: 26 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2026-02-13T01:57:12.966 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:56:56.040+0000 7f857bb4c640 19 osd.1 pg_epoch: 26 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2026-02-13T01:57:12.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-13T01:57:12.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T01:57:12.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T01:57:12.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T01:57:12.967 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T01:57:12.967 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T01:57:12.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T01:57:12.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T01:57:12.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T01:57:13.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T01:57:13.077 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T01:57:13.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T01:57:13.078 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T01:57:13.079 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T01:57:13.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T01:57:13.080 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T01:57:13.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T01:57:13.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T01:57:13.082 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T01:57:13.082 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T01:57:13.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T01:57:13.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T01:57:13.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T01:57:13.098 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T01:57:13.098 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:57:13.098 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:57:13.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T01:57:13.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T01:57:13.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T01:57:13.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-13T01:57:13.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-13T01:57:13.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-13T01:57:13.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-13T01:57:13.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T01:57:13.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T01:57:13.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T01:57:13.101 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T01:57:13.101 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T01:57:13.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T01:57:13.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T01:57:13.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T01:57:13.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T01:57:13.104 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T01:57:13.104 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T01:57:13.105 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T01:57:13.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T01:57:13.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T01:57:13.106 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T01:57:13.107 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T01:57:13.107 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T01:57:13.108 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T01:57:13.108 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T01:57:13.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T01:57:13.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T01:57:13.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T01:57:13.111 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T01:57:13.111 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:57:13.111 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:57:13.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T01:57:13.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T01:57:13.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T01:57:13.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-13T01:57:13.115 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T01:57:13.115 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:57:13.115 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:57:13.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.35207 2026-02-13T01:57:13.117 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T01:57:13.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-13T01:57:13.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T01:57:13.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-13T01:57:13.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_failed td/osd-scrub-repair 2026-02-13T01:57:13.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:598: TEST_auto_repair_bluestore_failed: local dir=td/osd-scrub-repair 2026-02-13T01:57:13.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:599: TEST_auto_repair_bluestore_failed: local poolname=testpool 2026-02-13T01:57:13.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:602: TEST_auto_repair_bluestore_failed: run_mon td/osd-scrub-repair a 2026-02-13T01:57:13.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-13T01:57:13.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-13T01:57:13.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-13T01:57:13.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-13T01:57:13.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-13T01:57:13.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-13T01:57:13.141 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-13T01:57:13.141 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:57:13.141 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:57:13.141 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:57:13.141 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:57:13.141 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:57:13.142 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:57:13.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-13T01:57:13.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-13T01:57:13.180 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-13T01:57:13.181 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T01:57:13.181 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T01:57:13.181 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-13T01:57:13.182 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-13T01:57:13.182 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T01:57:13.182 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T01:57:13.182 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T01:57:13.182 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T01:57:13.182 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:57:13.182 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:57:13.183 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T01:57:13.183 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T01:57:13.183 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get fsid 2026-02-13T01:57:13.239 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-13T01:57:13.239 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T01:57:13.239 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T01:57:13.239 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-13T01:57:13.240 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-13T01:57:13.240 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T01:57:13.240 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T01:57:13.241 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T01:57:13.241 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T01:57:13.241 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:57:13.241 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:57:13.241 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T01:57:13.241 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T01:57:13.242 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get mon_host 2026-02-13T01:57:13.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:603: TEST_auto_repair_bluestore_failed: run_mgr td/osd-scrub-repair x 2026-02-13T01:57:13.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-13T01:57:13.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-13T01:57:13.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-13T01:57:13.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-13T01:57:13.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-13T01:57:13.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-13T01:57:13.392 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-13T01:57:13.392 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:57:13.392 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:57:13.392 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:57:13.392 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:57:13.393 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:57:13.393 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:57:13.393 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T01:57:13.394 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T01:57:13.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:606: TEST_auto_repair_bluestore_failed: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-13T01:57:13.413 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:607: TEST_auto_repair_bluestore_failed: seq 0 2 2026-02-13T01:57:13.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:607: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2026-02-13T01:57:13.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:608: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T01:57:13.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T01:57:13.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T01:57:13.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-13T01:57:13.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T01:57:13.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T01:57:13.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T01:57:13.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T01:57:13.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T01:57:13.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T01:57:13.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T01:57:13.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T01:57:13.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T01:57:13.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T01:57:13.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T01:57:13.419 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T01:57:13.419 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:57:13.419 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:57:13.419 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:57:13.419 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:57:13.419 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:57:13.420 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:57:13.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:57:13.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T01:57:13.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T01:57:13.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T01:57:13.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T01:57:13.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T01:57:13.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T01:57:13.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T01:57:13.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T01:57:13.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T01:57:13.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T01:57:13.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-13T01:57:13.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T01:57:13.421 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T01:57:13.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=64a666f3-83f2-41ab-a900-898a1e7c223a 2026-02-13T01:57:13.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 64a666f3-83f2-41ab-a900-898a1e7c223a' 2026-02-13T01:57:13.422 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 64a666f3-83f2-41ab-a900-898a1e7c223a 2026-02-13T01:57:13.422 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T01:57:13.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD5hI5pydF5GRAAGf0tqGBpvVA4azBnPoC4ug== 2026-02-13T01:57:13.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD5hI5pydF5GRAAGf0tqGBpvVA4azBnPoC4ug=="}' 2026-02-13T01:57:13.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 64a666f3-83f2-41ab-a900-898a1e7c223a -i td/osd-scrub-repair/0/new.json 2026-02-13T01:57:13.520 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T01:57:13.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-13T01:57:13.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQD5hI5pydF5GRAAGf0tqGBpvVA4azBnPoC4ug== --osd-uuid 64a666f3-83f2-41ab-a900-898a1e7c223a 2026-02-13T01:57:13.546 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:13.540+0000 7fe3135d2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:57:13.547 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:13.540+0000 7fe3135d2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:57:13.548 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:13.544+0000 7fe3135d2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:57:13.549 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:13.544+0000 7fe3135d2a40 -1 bdev(0x564bc3118c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-13T01:57:13.549 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:13.544+0000 7fe3135d2a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-13T01:57:14.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-13T01:57:14.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T01:57:14.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-13T01:57:14.293 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T01:57:14.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T01:57:14.389 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T01:57:14.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-13T01:57:14.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T01:57:14.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T01:57:14.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T01:57:14.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T01:57:14.413 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:14.408+0000 7fb14b967a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:57:14.415 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:14.408+0000 7fb14b967a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:57:14.417 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:14.412+0000 7fb14b967a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:57:14.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-13T01:57:14.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T01:57:14.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T01:57:14.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T01:57:14.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T01:57:14.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:57:14.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T01:57:14.485 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T01:57:14.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:57:14.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T01:57:14.576 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:57:14.817 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:14.812+0000 7fb14b967a40 -1 Falling back to public interface 2026-02-13T01:57:15.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:57:15.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:57:15.579 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T01:57:15.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T01:57:15.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:57:15.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T01:57:15.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:57:15.833 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:15.828+0000 7fb14b967a40 -1 osd.0 0 log_to_monitors true 2026-02-13T01:57:16.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:57:16.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:57:16.673 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T01:57:16.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T01:57:16.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:57:16.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T01:57:16.836 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3526058320,v1:127.0.0.1:6801/3526058320] [v2:127.0.0.1:6802/3526058320,v1:127.0.0.1:6803/3526058320] exists,up 64a666f3-83f2-41ab-a900-898a1e7c223a 2026-02-13T01:57:16.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T01:57:16.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T01:57:16.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T01:57:16.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:607: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2026-02-13T01:57:16.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:608: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T01:57:16.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T01:57:16.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T01:57:16.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-13T01:57:16.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T01:57:16.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T01:57:16.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T01:57:16.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T01:57:16.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T01:57:16.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T01:57:16.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T01:57:16.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T01:57:16.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T01:57:16.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T01:57:16.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T01:57:16.841 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T01:57:16.841 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:57:16.842 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:57:16.842 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:57:16.842 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:57:16.842 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:57:16.842 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:57:16.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:57:16.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T01:57:16.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T01:57:16.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T01:57:16.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T01:57:16.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T01:57:16.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T01:57:16.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T01:57:16.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T01:57:16.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T01:57:16.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T01:57:16.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-13T01:57:16.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T01:57:16.843 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T01:57:16.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1cfa0d8f-b9d5-4166-8842-6f71c5d0882d 2026-02-13T01:57:16.844 INFO:tasks.workunit.client.0.trial112.stdout:add osd1 1cfa0d8f-b9d5-4166-8842-6f71c5d0882d 2026-02-13T01:57:16.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 1cfa0d8f-b9d5-4166-8842-6f71c5d0882d' 2026-02-13T01:57:16.844 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T01:57:16.854 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD8hI5p8vDGMhAAvcmljSwux5Wy9JYhfeDj8w== 2026-02-13T01:57:16.854 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD8hI5p8vDGMhAAvcmljSwux5Wy9JYhfeDj8w=="}' 2026-02-13T01:57:16.854 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1cfa0d8f-b9d5-4166-8842-6f71c5d0882d -i td/osd-scrub-repair/1/new.json 2026-02-13T01:57:17.022 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T01:57:17.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-13T01:57:17.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQD8hI5p8vDGMhAAvcmljSwux5Wy9JYhfeDj8w== --osd-uuid 1cfa0d8f-b9d5-4166-8842-6f71c5d0882d 2026-02-13T01:57:17.051 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:17.044+0000 7feec0cc5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:57:17.052 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:17.048+0000 7feec0cc5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:57:17.053 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:17.048+0000 7feec0cc5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:57:17.054 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:17.048+0000 7feec0cc5a40 -1 bdev(0x5622d25eac00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-13T01:57:17.054 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:17.048+0000 7feec0cc5a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-13T01:57:17.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-13T01:57:17.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T01:57:17.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-13T01:57:17.696 INFO:tasks.workunit.client.0.trial112.stdout:adding osd1 key to auth repository 2026-02-13T01:57:17.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T01:57:17.906 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T01:57:17.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-13T01:57:17.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T01:57:17.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T01:57:17.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T01:57:17.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T01:57:17.931 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:17.924+0000 7f7ac9171a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:57:17.933 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:17.928+0000 7f7ac9171a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:57:17.934 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:17.928+0000 7f7ac9171a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:57:18.066 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-13T01:57:18.066 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T01:57:18.066 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T01:57:18.066 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T01:57:18.066 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T01:57:18.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:57:18.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T01:57:18.067 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T01:57:18.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:57:18.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T01:57:18.230 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:57:18.309 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:18.304+0000 7f7ac9171a40 -1 Falling back to public interface 2026-02-13T01:57:18.954 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:18.948+0000 7f7ac9171a40 -1 osd.1 0 log_to_monitors true 2026-02-13T01:57:19.232 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:57:19.232 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:57:19.232 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T01:57:19.233 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T01:57:19.233 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:57:19.233 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T01:57:19.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:57:20.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:57:20.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:57:20.389 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T01:57:20.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T01:57:20.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:57:20.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T01:57:20.549 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/442269876,v1:127.0.0.1:6811/442269876] [v2:127.0.0.1:6812/442269876,v1:127.0.0.1:6813/442269876] exists,up 1cfa0d8f-b9d5-4166-8842-6f71c5d0882d 2026-02-13T01:57:20.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T01:57:20.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T01:57:20.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T01:57:20.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:607: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2026-02-13T01:57:20.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:608: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T01:57:20.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T01:57:20.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T01:57:20.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-13T01:57:20.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T01:57:20.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-13T01:57:20.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T01:57:20.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T01:57:20.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T01:57:20.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T01:57:20.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-13T01:57:20.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-13T01:57:20.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T01:57:20.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T01:57:20.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T01:57:20.554 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T01:57:20.554 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:57:20.554 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:57:20.554 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:57:20.554 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:57:20.554 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:57:20.554 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:57:20.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:57:20.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T01:57:20.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T01:57:20.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T01:57:20.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T01:57:20.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T01:57:20.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T01:57:20.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T01:57:20.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T01:57:20.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T01:57:20.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T01:57:20.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-13T01:57:20.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-13T01:57:20.556 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T01:57:20.556 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ff2e978e-ef6d-41ea-9e09-f44fd18d71f0 2026-02-13T01:57:20.556 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 ff2e978e-ef6d-41ea-9e09-f44fd18d71f0' 2026-02-13T01:57:20.557 INFO:tasks.workunit.client.0.trial112.stdout:add osd2 ff2e978e-ef6d-41ea-9e09-f44fd18d71f0 2026-02-13T01:57:20.557 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T01:57:20.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAAhY5pJ+ikIRAA7aA2NTgFNWElIPGUKIKZOA== 2026-02-13T01:57:20.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAAhY5pJ+ikIRAA7aA2NTgFNWElIPGUKIKZOA=="}' 2026-02-13T01:57:20.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ff2e978e-ef6d-41ea-9e09-f44fd18d71f0 -i td/osd-scrub-repair/2/new.json 2026-02-13T01:57:20.724 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T01:57:20.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-13T01:57:20.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAAhY5pJ+ikIRAA7aA2NTgFNWElIPGUKIKZOA== --osd-uuid ff2e978e-ef6d-41ea-9e09-f44fd18d71f0 2026-02-13T01:57:20.759 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:20.752+0000 7f5d715e2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:57:20.761 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:20.756+0000 7f5d715e2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:57:20.762 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:20.756+0000 7f5d715e2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:57:20.762 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:20.756+0000 7f5d715e2a40 -1 bdev(0x560ed3ff0c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-13T01:57:20.763 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:20.756+0000 7f5d715e2a40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-13T01:57:21.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-13T01:57:21.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T01:57:21.376 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-13T01:57:21.376 INFO:tasks.workunit.client.0.trial112.stdout:adding osd2 key to auth repository 2026-02-13T01:57:21.376 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T01:57:21.573 INFO:tasks.workunit.client.0.trial112.stdout:start osd.2 2026-02-13T01:57:21.574 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-13T01:57:21.575 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T01:57:21.575 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T01:57:21.575 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T01:57:21.575 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T01:57:21.591 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:21.584+0000 7fe575881a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:57:21.592 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:21.588+0000 7fe575881a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:57:21.593 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:21.588+0000 7fe575881a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:57:21.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-13T01:57:21.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T01:57:21.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T01:57:21.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T01:57:21.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T01:57:21.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:57:21.731 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T01:57:21.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T01:57:21.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:57:21.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T01:57:21.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:57:22.038 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:22.032+0000 7fe575881a40 -1 Falling back to public interface 2026-02-13T01:57:22.642 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:22.636+0000 7fe575881a40 -1 osd.2 0 log_to_monitors true 2026-02-13T01:57:22.901 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:57:22.901 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:57:22.902 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T01:57:22.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T01:57:22.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:57:22.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T01:57:23.064 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:57:24.066 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:57:24.066 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:57:24.067 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T01:57:24.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T01:57:24.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:57:24.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T01:57:24.226 INFO:tasks.workunit.client.0.trial112.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2255790764,v1:127.0.0.1:6819/2255790764] [v2:127.0.0.1:6820/2255790764,v1:127.0.0.1:6821/2255790764] exists,up ff2e978e-ef6d-41ea-9e09-f44fd18d71f0 2026-02-13T01:57:24.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T01:57:24.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T01:57:24.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T01:57:24.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:611: TEST_auto_repair_bluestore_failed: create_pool testpool 1 1 2026-02-13T01:57:24.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create testpool 1 1 2026-02-13T01:57:24.433 INFO:tasks.workunit.client.0.trial112.stderr:pool 'testpool' created 2026-02-13T01:57:24.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T01:57:25.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:612: TEST_auto_repair_bluestore_failed: ceph osd pool set testpool size 2 2026-02-13T01:57:25.650 INFO:tasks.workunit.client.0.trial112.stderr:set pool 1 size to 2 2026-02-13T01:57:25.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed: wait_for_clean 2026-02-13T01:57:25.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T01:57:25.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T01:57:25.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T01:57:25.663 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T01:57:25.663 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T01:57:25.663 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T01:57:25.664 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T01:57:25.664 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T01:57:25.664 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T01:57:25.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T01:57:25.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T01:57:25.773 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T01:57:25.773 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T01:57:25.773 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T01:57:25.773 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T01:57:25.925 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T01:57:25.925 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:57:25.926 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T01:57:25.926 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T01:57:25.926 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:57:25.926 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T01:57:26.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-13T01:57:26.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-13T01:57:26.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-13T01:57:26.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:57:26.000 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T01:57:26.074 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705667 2026-02-13T01:57:26.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705667 2026-02-13T01:57:26.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705667' 2026-02-13T01:57:26.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:57:26.075 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T01:57:26.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542146 2026-02-13T01:57:26.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542146 2026-02-13T01:57:26.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705667 2-60129542146' 2026-02-13T01:57:26.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:57:26.146 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-13T01:57:26.147 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:57:26.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T01:57:26.148 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-13T01:57:26.149 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:57:26.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-13T01:57:26.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-13T01:57:26.150 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836483 2026-02-13T01:57:26.151 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:57:26.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-13T01:57:26.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:57:27.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T01:57:27.315 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:57:27.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-13T01:57:27.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:57:28.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T01:57:28.481 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:57:28.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836483 2026-02-13T01:57:28.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:57:28.639 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705667 2026-02-13T01:57:28.639 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:57:28.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T01:57:28.641 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705667 2026-02-13T01:57:28.641 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:57:28.642 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705667 2026-02-13T01:57:28.643 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705667 2026-02-13T01:57:28.643 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705667' 2026-02-13T01:57:28.643 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T01:57:28.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705667 -lt 38654705667 2026-02-13T01:57:28.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:57:28.810 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542146 2026-02-13T01:57:28.810 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:57:28.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T01:57:28.812 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542146 2026-02-13T01:57:28.813 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:57:28.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542146 2026-02-13T01:57:28.814 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542146 2026-02-13T01:57:28.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542146' 2026-02-13T01:57:28.815 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T01:57:28.988 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542146 -lt 60129542146 2026-02-13T01:57:28.988 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T01:57:28.989 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:57:28.989 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:57:29.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T01:57:29.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T01:57:29.189 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T01:57:29.189 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T01:57:29.189 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T01:57:29.189 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T01:57:29.190 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T01:57:29.190 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T01:57:29.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-13T01:57:29.356 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T01:57:29.356 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:57:29.356 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:57:29.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-13T01:57:29.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' -1 2026-02-13T01:57:29.556 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-13T01:57:29.556 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=0 2026-02-13T01:57:29.556 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-13T01:57:29.556 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-13T01:57:29.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-13T01:57:29.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T01:57:29.657 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T01:57:29.657 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T01:57:29.657 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T01:57:29.657 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T01:57:29.658 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T01:57:29.658 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T01:57:29.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-13T01:57:29.822 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T01:57:29.822 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:57:29.822 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:57:30.021 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-13T01:57:30.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-13T01:57:30.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-13T01:57:30.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-13T01:57:30.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-13T01:57:30.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-13T01:57:30.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-13T01:57:30.022 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-13T01:57:30.023 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-13T01:57:30.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-13T01:57:30.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-13T01:57:30.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-13T01:57:30.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-13T01:57:30.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-13T01:57:30.416 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-13T01:57:30.416 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T01:57:30.417 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T01:57:30.417 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T01:57:30.418 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T01:57:30.418 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T01:57:30.418 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T01:57:30.418 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T01:57:30.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-13T01:57:30.579 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T01:57:30.579 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:57:30.579 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:57:30.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-13T01:57:30.773 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-13T01:57:30.773 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-13T01:57:30.773 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-13T01:57:30.773 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-13T01:57:30.773 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-13T01:57:30.773 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-13T01:57:30.774 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-13T01:57:30.774 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-13T01:57:30.970 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-13T01:57:30.970 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-13T01:57:30.970 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 2 >= 13 )) 2026-02-13T01:57:30.970 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-13T01:57:30.971 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.4 2026-02-13T01:57:31.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-13T01:57:31.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T01:57:31.372 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T01:57:31.372 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T01:57:31.372 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T01:57:31.373 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T01:57:31.373 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T01:57:31.373 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T01:57:31.536 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-13T01:57:31.537 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T01:57:31.537 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:57:31.537 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:57:31.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-13T01:57:31.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-13T01:57:31.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-13T01:57:31.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-13T01:57:31.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-13T01:57:31.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-13T01:57:31.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-13T01:57:31.732 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-13T01:57:31.733 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-13T01:57:31.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-13T01:57:31.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-13T01:57:31.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 3 >= 13 )) 2026-02-13T01:57:31.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-13T01:57:31.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.8 2026-02-13T01:57:32.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-13T01:57:32.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T01:57:32.732 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T01:57:32.732 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T01:57:32.732 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T01:57:32.732 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T01:57:32.732 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T01:57:32.732 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T01:57:32.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T01:57:32.895 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T01:57:32.896 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:57:32.896 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:57:33.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T01:57:33.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T01:57:33.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T01:57:33.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:616: TEST_auto_repair_bluestore_failed: local payload=ABCDEF 2026-02-13T01:57:33.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:617: TEST_auto_repair_bluestore_failed: echo ABCDEF 2026-02-13T01:57:33.096 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed: seq 1 10 2026-02-13T01:57:33.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2026-02-13T01:57:33.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:620: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2026-02-13T01:57:33.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2026-02-13T01:57:33.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:620: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2026-02-13T01:57:33.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2026-02-13T01:57:33.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:620: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2026-02-13T01:57:33.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2026-02-13T01:57:33.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:620: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2026-02-13T01:57:33.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2026-02-13T01:57:33.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:620: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2026-02-13T01:57:33.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2026-02-13T01:57:33.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:620: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2026-02-13T01:57:33.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2026-02-13T01:57:33.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:620: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2026-02-13T01:57:33.270 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2026-02-13T01:57:33.270 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:620: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2026-02-13T01:57:33.295 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2026-02-13T01:57:33.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:620: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2026-02-13T01:57:33.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2026-02-13T01:57:33.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:620: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2026-02-13T01:57:33.347 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:625: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2026-02-13T01:57:33.347 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-13T01:57:33.347 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-13T01:57:33.348 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2026-02-13T01:57:33.348 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-13T01:57:33.348 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-13T01:57:33.348 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-13T01:57:33.348 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T01:57:33.512 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-13T01:57:33.513 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2026-02-13T01:57:33.513 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-13T01:57:33.681 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:625: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2026-02-13T01:57:33.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T01:57:33.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T01:57:33.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T01:57:33.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T01:57:33.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2026-02-13T01:57:33.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T01:57:33.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T01:57:33.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T01:57:33.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T01:57:33.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T01:57:33.683 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T01:57:33.683 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T01:57:33.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T01:57:33.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T01:57:33.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T01:57:33.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T01:57:33.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2026-02-13T01:57:33.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T01:57:33.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T01:57:33.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T01:57:33.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T01:57:33.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T01:57:33.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2026-02-13T01:57:34.172 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:ff7b1f36:::obj1:head# 2026-02-13T01:57:34.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T01:57:34.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T01:57:34.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T01:57:34.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T01:57:34.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T01:57:34.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T01:57:34.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T01:57:34.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T01:57:34.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T01:57:34.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T01:57:34.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T01:57:34.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T01:57:34.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T01:57:34.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T01:57:34.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T01:57:34.275 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T01:57:34.275 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:57:34.276 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:57:34.276 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:57:34.276 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:57:34.276 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:57:34.276 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:57:34.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:57:34.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T01:57:34.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T01:57:34.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T01:57:34.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T01:57:34.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T01:57:34.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T01:57:34.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T01:57:34.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T01:57:34.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-13T01:57:34.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T01:57:34.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T01:57:34.279 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T01:57:34.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T01:57:34.279 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T01:57:34.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T01:57:34.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T01:57:34.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T01:57:34.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T01:57:34.298 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:34.292+0000 7f1f12e2ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:57:34.300 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:34.292+0000 7f1f12e2ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:57:34.301 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:34.296+0000 7f1f12e2ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:57:34.439 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T01:57:34.439 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T01:57:34.439 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T01:57:34.439 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T01:57:34.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T01:57:34.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:57:34.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T01:57:34.440 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T01:57:34.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:57:34.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T01:57:34.598 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:57:34.694 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:34.688+0000 7f1f12e2ea40 -1 Falling back to public interface 2026-02-13T01:57:35.279 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:35.272+0000 7f1f12e2ea40 -1 osd.0 21 log_to_monitors true 2026-02-13T01:57:35.600 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:57:35.600 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:57:35.600 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T01:57:35.600 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:57:35.600 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:57:35.600 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T01:57:35.759 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 21 last_clean_interval [5,21) [v2:127.0.0.1:6800/4067154335,v1:127.0.0.1:6801/4067154335] [v2:127.0.0.1:6802/4067154335,v1:127.0.0.1:6803/4067154335] exists,up 64a666f3-83f2-41ab-a900-898a1e7c223a 2026-02-13T01:57:35.760 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T01:57:35.760 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T01:57:35.760 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T01:57:35.760 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T01:57:35.760 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T01:57:35.760 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T01:57:35.760 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T01:57:35.760 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T01:57:35.761 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T01:57:35.761 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T01:57:35.761 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T01:57:35.761 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T01:57:35.761 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T01:57:35.859 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T01:57:35.859 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T01:57:35.859 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T01:57:35.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T01:57:35.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T01:57:35.860 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T01:57:36.024 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T01:57:36.024 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:57:36.025 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T01:57:36.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T01:57:36.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:57:36.025 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T01:57:36.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215106 2026-02-13T01:57:36.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215106 2026-02-13T01:57:36.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215106' 2026-02-13T01:57:36.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:57:36.103 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T01:57:36.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705670 2026-02-13T01:57:36.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705670 2026-02-13T01:57:36.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215106 1-38654705670' 2026-02-13T01:57:36.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:57:36.177 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T01:57:36.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542149 2026-02-13T01:57:36.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542149 2026-02-13T01:57:36.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215106 1-38654705670 2-60129542149' 2026-02-13T01:57:36.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:57:36.249 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-103079215106 2026-02-13T01:57:36.249 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:57:36.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T01:57:36.250 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-103079215106 2026-02-13T01:57:36.251 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:57:36.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215106 2026-02-13T01:57:36.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2026-02-13T01:57:36.252 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 103079215106 2026-02-13T01:57:36.253 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:57:36.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 103079215106 2026-02-13T01:57:36.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:57:37.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T01:57:37.420 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:57:37.576 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 103079215106 2026-02-13T01:57:37.576 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:57:38.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T01:57:38.577 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:57:38.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215106 -lt 103079215106 2026-02-13T01:57:38.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:57:38.743 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705670 2026-02-13T01:57:38.743 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:57:38.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T01:57:38.745 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705670 2026-02-13T01:57:38.745 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:57:38.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705670 2026-02-13T01:57:38.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705670' 2026-02-13T01:57:38.746 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 38654705670 2026-02-13T01:57:38.747 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T01:57:38.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705670 -lt 38654705670 2026-02-13T01:57:38.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:57:38.913 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542149 2026-02-13T01:57:38.913 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:57:38.914 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T01:57:38.915 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542149 2026-02-13T01:57:38.915 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:57:38.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542149 2026-02-13T01:57:38.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542149' 2026-02-13T01:57:38.917 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 60129542149 2026-02-13T01:57:38.917 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T01:57:39.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542149 -lt 60129542149 2026-02-13T01:57:39.079 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T01:57:39.079 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:57:39.080 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:57:39.280 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T01:57:39.280 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T01:57:39.280 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T01:57:39.280 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T01:57:39.280 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T01:57:39.281 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T01:57:39.281 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T01:57:39.281 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T01:57:39.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T01:57:39.441 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T01:57:39.441 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:57:39.441 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:57:39.642 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T01:57:39.643 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T01:57:39.643 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T01:57:39.643 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:627: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2026-02-13T01:57:39.643 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-13T01:57:39.643 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-13T01:57:39.644 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2026-02-13T01:57:39.644 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-13T01:57:39.644 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-13T01:57:39.644 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-13T01:57:39.644 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T01:57:39.808 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-13T01:57:39.809 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2026-02-13T01:57:39.809 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-13T01:57:39.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:627: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2026-02-13T01:57:39.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T01:57:39.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T01:57:39.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T01:57:39.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T01:57:39.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2026-02-13T01:57:39.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T01:57:39.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T01:57:39.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T01:57:39.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T01:57:39.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T01:57:39.976 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T01:57:39.976 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T01:57:39.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T01:57:39.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T01:57:39.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T01:57:40.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T01:57:40.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2026-02-13T01:57:40.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T01:57:40.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T01:57:40.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T01:57:40.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T01:57:40.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T01:57:40.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2026-02-13T01:57:40.472 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:104778fc:::obj2:head# 2026-02-13T01:57:40.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T01:57:40.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T01:57:40.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T01:57:40.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T01:57:40.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T01:57:40.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T01:57:40.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T01:57:40.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T01:57:40.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T01:57:40.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T01:57:40.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T01:57:40.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T01:57:40.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T01:57:40.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T01:57:40.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T01:57:40.564 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T01:57:40.564 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:57:40.564 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:57:40.564 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:57:40.564 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:57:40.564 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:57:40.564 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:57:40.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:57:40.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T01:57:40.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T01:57:40.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T01:57:40.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T01:57:40.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T01:57:40.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T01:57:40.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T01:57:40.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T01:57:40.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-13T01:57:40.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T01:57:40.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T01:57:40.566 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T01:57:40.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T01:57:40.567 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T01:57:40.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T01:57:40.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T01:57:40.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T01:57:40.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T01:57:40.589 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:40.584+0000 7fddd26e3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:57:40.590 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:40.584+0000 7fddd26e3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:57:40.592 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:40.584+0000 7fddd26e3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:57:40.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T01:57:40.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T01:57:40.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T01:57:40.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T01:57:40.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T01:57:40.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:57:40.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T01:57:40.732 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T01:57:40.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:57:40.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T01:57:40.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:57:41.010 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:41.004+0000 7fddd26e3a40 -1 Falling back to public interface 2026-02-13T01:57:41.610 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:41.604+0000 7fddd26e3a40 -1 osd.0 26 log_to_monitors true 2026-02-13T01:57:41.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:57:41.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:57:41.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T01:57:41.897 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:57:41.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:57:41.898 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T01:57:42.053 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [24,26) [v2:127.0.0.1:6800/2028601013,v1:127.0.0.1:6801/2028601013] [v2:127.0.0.1:6802/2028601013,v1:127.0.0.1:6803/2028601013] exists,up 64a666f3-83f2-41ab-a900-898a1e7c223a 2026-02-13T01:57:42.053 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T01:57:42.053 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T01:57:42.053 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T01:57:42.054 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T01:57:42.054 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T01:57:42.054 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T01:57:42.054 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T01:57:42.054 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T01:57:42.055 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T01:57:42.055 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T01:57:42.055 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T01:57:42.055 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T01:57:42.055 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T01:57:42.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T01:57:42.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T01:57:42.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T01:57:42.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T01:57:42.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T01:57:42.156 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T01:57:42.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T01:57:42.314 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:57:42.314 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T01:57:42.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T01:57:42.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:57:42.314 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T01:57:42.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018882 2026-02-13T01:57:42.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018882 2026-02-13T01:57:42.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018882' 2026-02-13T01:57:42.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:57:42.391 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T01:57:42.465 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705672 2026-02-13T01:57:42.465 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705672 2026-02-13T01:57:42.465 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018882 1-38654705672' 2026-02-13T01:57:42.465 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:57:42.465 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T01:57:42.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542151 2026-02-13T01:57:42.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542151 2026-02-13T01:57:42.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018882 1-38654705672 2-60129542151' 2026-02-13T01:57:42.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:57:42.539 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-128849018882 2026-02-13T01:57:42.539 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:57:42.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T01:57:42.540 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-128849018882 2026-02-13T01:57:42.540 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:57:42.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018882 2026-02-13T01:57:42.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2026-02-13T01:57:42.542 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 128849018882 2026-02-13T01:57:42.542 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:57:42.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215106 -lt 128849018882 2026-02-13T01:57:42.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:57:43.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T01:57:43.713 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:57:43.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018882 -lt 128849018882 2026-02-13T01:57:43.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:57:43.868 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705672 2026-02-13T01:57:43.868 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:57:43.870 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T01:57:43.870 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705672 2026-02-13T01:57:43.871 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:57:43.872 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705672 2026-02-13T01:57:43.872 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705672' 2026-02-13T01:57:43.872 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 38654705672 2026-02-13T01:57:43.872 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T01:57:44.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705672 -lt 38654705672 2026-02-13T01:57:44.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:57:44.033 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542151 2026-02-13T01:57:44.033 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:57:44.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T01:57:44.035 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542151 2026-02-13T01:57:44.035 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:57:44.037 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542151 2026-02-13T01:57:44.037 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542151' 2026-02-13T01:57:44.037 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 60129542151 2026-02-13T01:57:44.037 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T01:57:44.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542151 -lt 60129542151 2026-02-13T01:57:44.194 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T01:57:44.195 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:57:44.195 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:57:44.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T01:57:44.385 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T01:57:44.385 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T01:57:44.385 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T01:57:44.385 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T01:57:44.385 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T01:57:44.385 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T01:57:44.385 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T01:57:44.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T01:57:44.547 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T01:57:44.547 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:57:44.548 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:57:44.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T01:57:44.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T01:57:44.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T01:57:44.754 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:628: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2026-02-13T01:57:44.754 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-13T01:57:44.754 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-13T01:57:44.754 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-13T01:57:44.754 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T01:57:44.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:628: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2026-02-13T01:57:44.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T01:57:44.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T01:57:44.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T01:57:44.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T01:57:44.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2026-02-13T01:57:44.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T01:57:44.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T01:57:44.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T01:57:44.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T01:57:44.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T01:57:44.917 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T01:57:44.917 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T01:57:44.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T01:57:44.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T01:57:44.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T01:57:45.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T01:57:45.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2026-02-13T01:57:45.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T01:57:45.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T01:57:45.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T01:57:45.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T01:57:45.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T01:57:45.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2026-02-13T01:57:45.527 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T01:57:45.527 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T01:57:45.527 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T01:57:45.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T01:57:45.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T01:57:45.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T01:57:45.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T01:57:45.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T01:57:45.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T01:57:45.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T01:57:45.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T01:57:45.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T01:57:45.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T01:57:45.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T01:57:45.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T01:57:45.529 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T01:57:45.529 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:57:45.529 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:57:45.529 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:57:45.529 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:57:45.529 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:57:45.529 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:57:45.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:57:45.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T01:57:45.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T01:57:45.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T01:57:45.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T01:57:45.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T01:57:45.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T01:57:45.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T01:57:45.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T01:57:45.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-13T01:57:45.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T01:57:45.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T01:57:45.530 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T01:57:45.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T01:57:45.531 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T01:57:45.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T01:57:45.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T01:57:45.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T01:57:45.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T01:57:45.547 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:45.540+0000 7faa8c750a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:57:45.549 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:45.544+0000 7faa8c750a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:57:45.550 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:45.544+0000 7faa8c750a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:57:45.692 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T01:57:45.692 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T01:57:45.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T01:57:45.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T01:57:45.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T01:57:45.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:57:45.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T01:57:45.693 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T01:57:45.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:57:45.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T01:57:45.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:57:45.926 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:45.920+0000 7faa8c750a40 -1 Falling back to public interface 2026-02-13T01:57:46.537 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:57:46.532+0000 7faa8c750a40 -1 osd.1 32 log_to_monitors true 2026-02-13T01:57:46.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:57:46.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:57:46.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T01:57:46.853 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:57:46.854 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:57:46.854 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T01:57:47.011 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 35 up_thru 35 down_at 32 last_clean_interval [9,32) [v2:127.0.0.1:6810/2252159031,v1:127.0.0.1:6811/2252159031] [v2:127.0.0.1:6812/2252159031,v1:127.0.0.1:6813/2252159031] exists,up 1cfa0d8f-b9d5-4166-8842-6f71c5d0882d 2026-02-13T01:57:47.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T01:57:47.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T01:57:47.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T01:57:47.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T01:57:47.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T01:57:47.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T01:57:47.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T01:57:47.012 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T01:57:47.013 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T01:57:47.013 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T01:57:47.013 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T01:57:47.013 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T01:57:47.013 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T01:57:47.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T01:57:47.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T01:57:47.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T01:57:47.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T01:57:47.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T01:57:47.118 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T01:57:47.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T01:57:47.278 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:57:47.279 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T01:57:47.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T01:57:47.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:57:47.279 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T01:57:47.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018884 2026-02-13T01:57:47.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018884 2026-02-13T01:57:47.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018884' 2026-02-13T01:57:47.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:57:47.354 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T01:57:47.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855362 2026-02-13T01:57:47.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855362 2026-02-13T01:57:47.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018884 1-150323855362' 2026-02-13T01:57:47.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:57:47.428 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T01:57:47.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542153 2026-02-13T01:57:47.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542153 2026-02-13T01:57:47.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018884 1-150323855362 2-60129542153' 2026-02-13T01:57:47.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:57:47.502 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-128849018884 2026-02-13T01:57:47.502 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:57:47.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T01:57:47.503 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-128849018884 2026-02-13T01:57:47.503 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:57:47.505 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018884 2026-02-13T01:57:47.505 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 128849018884' 2026-02-13T01:57:47.505 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 128849018884 2026-02-13T01:57:47.506 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:57:47.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018884 -lt 128849018884 2026-02-13T01:57:47.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:57:47.663 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-150323855362 2026-02-13T01:57:47.663 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:57:47.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T01:57:47.665 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-150323855362 2026-02-13T01:57:47.665 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:57:47.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855362 2026-02-13T01:57:47.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 150323855362' 2026-02-13T01:57:47.667 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 150323855362 2026-02-13T01:57:47.667 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T01:57:47.839 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855362 -lt 150323855362 2026-02-13T01:57:47.839 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:57:47.839 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542153 2026-02-13T01:57:47.840 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:57:47.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T01:57:47.841 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542153 2026-02-13T01:57:47.842 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:57:47.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542153 2026-02-13T01:57:47.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542153' 2026-02-13T01:57:47.843 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 60129542153 2026-02-13T01:57:47.843 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T01:57:48.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542153 -lt 60129542153 2026-02-13T01:57:48.012 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T01:57:48.013 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:57:48.013 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:57:48.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T01:57:48.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T01:57:48.205 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T01:57:48.205 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T01:57:48.205 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T01:57:48.205 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T01:57:48.206 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T01:57:48.206 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T01:57:48.358 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T01:57:48.358 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T01:57:48.359 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:57:48.359 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:57:48.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T01:57:48.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T01:57:48.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T01:57:48.552 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:630: TEST_auto_repair_bluestore_failed: get_pg testpool obj1 2026-02-13T01:57:48.552 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2026-02-13T01:57:48.552 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=obj1 2026-02-13T01:57:48.552 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool obj1 2026-02-13T01:57:48.553 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T01:57:48.718 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:630: TEST_auto_repair_bluestore_failed: local pgid=1.0 2026-02-13T01:57:48.719 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:631: TEST_auto_repair_bluestore_failed: get_primary testpool obj1 2026-02-13T01:57:48.719 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-13T01:57:48.719 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-13T01:57:48.719 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2026-02-13T01:57:48.719 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T01:57:48.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:631: TEST_auto_repair_bluestore_failed: local primary=1 2026-02-13T01:57:48.885 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:632: TEST_auto_repair_bluestore_failed: get_last_scrub_stamp 1.0 2026-02-13T01:57:48.885 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T01:57:48.885 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T01:57:48.885 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T01:57:48.885 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T01:57:49.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:632: TEST_auto_repair_bluestore_failed: local last_scrub_stamp=2026-02-13T01:57:24.431303+0000 2026-02-13T01:57:49.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:633: TEST_auto_repair_bluestore_failed: ceph tell 1.0 schedule-deep-scrub 2026-02-13T01:57:49.111 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T01:57:49.112 INFO:tasks.workunit.client.0.trial112.stdout: "deep": true, 2026-02-13T01:57:49.112 INFO:tasks.workunit.client.0.trial112.stdout: "must": false, 2026-02-13T01:57:49.112 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "2026-02-01T21:09:39.110170+0000" 2026-02-13T01:57:49.112 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T01:57:49.122 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:636: TEST_auto_repair_bluestore_failed: wait_for_scrub 1.0 2026-02-13T01:57:24.431303+0000 2026-02-13T01:57:49.122 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-13T01:57:49.122 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T01:57:24.431303+0000 2026-02-13T01:57:49.122 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T01:57:49.122 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T01:57:49.122 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T01:57:49.123 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T01:57:49.123 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T01:57:49.124 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T01:57:49.124 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T01:57:49.124 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T01:57:49.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T01:57:24.431303+0000 '>' 2026-02-13T01:57:24.431303+0000 2026-02-13T01:57:49.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T01:57:50.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T01:57:50.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T01:57:50.288 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T01:57:50.289 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T01:57:50.289 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T01:57:50.289 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T01:57:50.289 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T01:57:50.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T01:57:24.431303+0000 '>' 2026-02-13T01:57:24.431303+0000 2026-02-13T01:57:50.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T01:57:51.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T01:57:51.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T01:57:51.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T01:57:51.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T01:57:51.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T01:57:51.456 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T01:57:51.456 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T01:57:51.619 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T01:57:24.431303+0000 '>' 2026-02-13T01:57:24.431303+0000 2026-02-13T01:57:51.620 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T01:57:52.620 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T01:57:52.621 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T01:57:52.621 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T01:57:52.621 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T01:57:52.622 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T01:57:52.622 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T01:57:52.622 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T01:57:52.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T01:57:24.431303+0000 '>' 2026-02-13T01:57:24.431303+0000 2026-02-13T01:57:52.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T01:57:53.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T01:57:53.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T01:57:53.787 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T01:57:53.787 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T01:57:53.787 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T01:57:53.787 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T01:57:53.787 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T01:57:53.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T01:57:24.431303+0000 '>' 2026-02-13T01:57:24.431303+0000 2026-02-13T01:57:53.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T01:57:54.950 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T01:57:54.950 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T01:57:54.951 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T01:57:54.951 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T01:57:54.951 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T01:57:54.952 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T01:57:54.952 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T01:57:55.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T01:57:24.431303+0000 '>' 2026-02-13T01:57:24.431303+0000 2026-02-13T01:57:55.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T01:57:56.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T01:57:56.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T01:57:56.113 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T01:57:56.113 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T01:57:56.114 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T01:57:56.114 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T01:57:56.114 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T01:57:56.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T01:57:24.431303+0000 '>' 2026-02-13T01:57:24.431303+0000 2026-02-13T01:57:56.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T01:57:57.277 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T01:57:57.277 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T01:57:57.278 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T01:57:57.278 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T01:57:57.278 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T01:57:57.278 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T01:57:57.279 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T01:57:57.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T01:57:24.431303+0000 '>' 2026-02-13T01:57:24.431303+0000 2026-02-13T01:57:57.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T01:57:58.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T01:57:58.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T01:57:58.445 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T01:57:58.445 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T01:57:58.446 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T01:57:58.446 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T01:57:58.446 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T01:57:58.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T01:57:55.519508+0000 '>' 2026-02-13T01:57:24.431303+0000 2026-02-13T01:57:58.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T01:57:58.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:637: TEST_auto_repair_bluestore_failed: wait_for_clean 2026-02-13T01:57:58.610 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T01:57:58.610 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T01:57:58.610 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T01:57:58.610 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T01:57:58.610 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T01:57:58.610 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T01:57:58.611 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T01:57:58.611 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T01:57:58.611 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T01:57:58.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T01:57:58.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T01:57:58.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T01:57:58.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T01:57:58.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T01:57:58.725 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T01:57:58.892 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T01:57:58.892 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:57:58.892 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T01:57:58.892 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T01:57:58.892 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:57:58.893 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T01:57:58.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018887 2026-02-13T01:57:58.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018887 2026-02-13T01:57:58.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018887' 2026-02-13T01:57:58.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:57:58.964 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T01:57:59.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855365 2026-02-13T01:57:59.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855365 2026-02-13T01:57:59.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018887 1-150323855365' 2026-02-13T01:57:59.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:57:59.036 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T01:57:59.107 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542157 2026-02-13T01:57:59.108 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542157 2026-02-13T01:57:59.108 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018887 1-150323855365 2-60129542157' 2026-02-13T01:57:59.108 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:57:59.108 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-128849018887 2026-02-13T01:57:59.109 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:57:59.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T01:57:59.110 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-128849018887 2026-02-13T01:57:59.110 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:57:59.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018887 2026-02-13T01:57:59.112 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 128849018887 2026-02-13T01:57:59.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 128849018887' 2026-02-13T01:57:59.112 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:57:59.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018886 -lt 128849018887 2026-02-13T01:57:59.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:58:00.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T01:58:00.276 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:58:00.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018887 -lt 128849018887 2026-02-13T01:58:00.439 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:58:00.439 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-150323855365 2026-02-13T01:58:00.439 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:58:00.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T01:58:00.441 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-150323855365 2026-02-13T01:58:00.441 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:58:00.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855365 2026-02-13T01:58:00.443 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 150323855365 2026-02-13T01:58:00.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 150323855365' 2026-02-13T01:58:00.444 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T01:58:00.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855365 -lt 150323855365 2026-02-13T01:58:00.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:58:00.606 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542157 2026-02-13T01:58:00.606 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:58:00.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T01:58:00.608 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542157 2026-02-13T01:58:00.608 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:58:00.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542157 2026-02-13T01:58:00.609 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542157 2026-02-13T01:58:00.610 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542157' 2026-02-13T01:58:00.610 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T01:58:00.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542157 -lt 60129542157 2026-02-13T01:58:00.784 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T01:58:00.785 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:58:00.785 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:58:00.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T01:58:00.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T01:58:00.985 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T01:58:00.985 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T01:58:00.985 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T01:58:00.985 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T01:58:00.985 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T01:58:00.985 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T01:58:01.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T01:58:01.148 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T01:58:01.149 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:58:01.149 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:58:01.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T01:58:01.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T01:58:01.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T01:58:01.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:638: TEST_auto_repair_bluestore_failed: flush_pg_stats 2026-02-13T01:58:01.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T01:58:01.350 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T01:58:01.509 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T01:58:01.509 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:58:01.509 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T01:58:01.509 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T01:58:01.509 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:58:01.509 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T01:58:01.584 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018888 2026-02-13T01:58:01.584 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018888 2026-02-13T01:58:01.584 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018888' 2026-02-13T01:58:01.584 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:58:01.584 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T01:58:01.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855367 2026-02-13T01:58:01.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855367 2026-02-13T01:58:01.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018888 1-150323855367' 2026-02-13T01:58:01.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:58:01.660 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T01:58:01.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542158 2026-02-13T01:58:01.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542158 2026-02-13T01:58:01.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018888 1-150323855367 2-60129542158' 2026-02-13T01:58:01.735 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:58:01.735 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-128849018888 2026-02-13T01:58:01.735 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:58:01.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T01:58:01.737 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-128849018888 2026-02-13T01:58:01.737 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:58:01.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018888 2026-02-13T01:58:01.739 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 128849018888 2026-02-13T01:58:01.739 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 128849018888' 2026-02-13T01:58:01.739 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:58:01.908 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018888 -lt 128849018888 2026-02-13T01:58:01.908 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:58:01.909 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-150323855367 2026-02-13T01:58:01.909 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:58:01.910 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T01:58:01.911 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-150323855367 2026-02-13T01:58:01.911 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:58:01.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855367 2026-02-13T01:58:01.913 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 150323855367 2026-02-13T01:58:01.913 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 150323855367' 2026-02-13T01:58:01.913 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T01:58:02.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855366 -lt 150323855367 2026-02-13T01:58:02.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:58:03.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T01:58:03.077 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T01:58:03.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855366 -lt 150323855367 2026-02-13T01:58:03.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:58:04.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T01:58:04.240 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T01:58:04.410 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855367 -lt 150323855367 2026-02-13T01:58:04.410 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:58:04.410 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542158 2026-02-13T01:58:04.411 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:58:04.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T01:58:04.413 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542158 2026-02-13T01:58:04.413 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:58:04.414 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542158 2026-02-13T01:58:04.414 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542158 2026-02-13T01:58:04.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542158' 2026-02-13T01:58:04.415 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T01:58:04.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542158 -lt 60129542158 2026-02-13T01:58:04.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:639: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2026-02-13T01:58:04.591 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:57:54.492+0000 7faa6ded5640 10 osd.1 pg_epoch: 36 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep(periodic-regular,auto-repair)] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2026-02-13T01:58:04.591 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:57:54.492+0000 7faa6ded5640 10 osd.1 pg_epoch: 36 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep(periodic-regular,auto-repair)] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2026-02-13T01:58:04.591 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:57:54.492+0000 7faa6ded5640 15 osd.1 pg_epoch: 36 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=20'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2026-02-13T01:58:04.591 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:57:54.492+0000 7faa6ded5640 20 osd.1 pg_epoch: 36 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=20'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish: setting for deep-scrub-after-repair (5 errors. 2 errors fixed) 2026-02-13T01:58:04.591 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:57:54.492+0000 7faa6ded5640 19 osd.1 pg_epoch: 36 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=20'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2026-02-13T01:58:04.591 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:57:55.515+0000 7faa69ecd640 10 osd.1 pg_epoch: 36 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent(after-repair,check-repair)] scrubber: scrub_finish before flags: CHECK_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2026-02-13T01:58:04.591 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:57:55.515+0000 7faa69ecd640 10 osd.1 pg_epoch: 36 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent(after-repair,check-repair)] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2026-02-13T01:58:04.591 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:57:55.515+0000 7faa69ecd640 19 osd.1 pg_epoch: 36 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent(after-repair,check-repair)] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2026-02-13T01:58:04.592 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:57:55.515+0000 7faa69ecd640 10 osd.1 pg_epoch: 36 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair(after-repair)] scrubber: scrub_finish 3 error(s) still present after re-scrub 2026-02-13T01:58:04.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:640: TEST_auto_repair_bluestore_failed: grep -q 'scrub_finish.*still present after re-scrub' td/osd-scrub-repair/osd.1.log 2026-02-13T01:58:04.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:641: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2026-02-13T01:58:04.743 INFO:tasks.workunit.client.0.trial112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-13T01:58:04.744 INFO:tasks.workunit.client.0.trial112.stdout:1.0 10 0 0 0 0 63 0 0 10 0 10 active+clean+inconsistent+failed_repair 2026-02-13T01:57:55.519574+0000 20'10 36:114 [1,0] 1 [1,0] 1 20'10 2026-02-13T01:57:55.519508+0000 20'10 2026-02-13T01:57:55.519508+0000 0 0 periodic scrub scheduled @ 2026-02-14T01:57:55.519508+0000 10 0 2026-02-13T01:58:04.744 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:58:04.744 INFO:tasks.workunit.client.0.trial112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-13T01:58:04.744 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T01:58:04.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:642: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2026-02-13T01:58:04.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:642: TEST_auto_repair_bluestore_failed: grep -q '^1.0.*+failed_repair' 2026-02-13T01:58:04.913 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T01:58:04.922 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:646: TEST_auto_repair_bluestore_failed: get_not_primary testpool obj1 2026-02-13T01:58:04.922 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-13T01:58:04.922 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=obj1 2026-02-13T01:58:04.924 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool obj1 2026-02-13T01:58:04.924 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-13T01:58:04.924 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-13T01:58:04.924 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2026-02-13T01:58:04.924 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T01:58:05.081 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-13T01:58:05.081 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool obj1 2026-02-13T01:58:05.081 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-13T01:58:05.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:646: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 list-attrs 2026-02-13T01:58:05.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T01:58:05.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T01:58:05.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T01:58:05.251 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T01:58:05.251 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 list-attrs 2026-02-13T01:58:05.251 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T01:58:05.251 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T01:58:05.251 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T01:58:05.251 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T01:58:05.251 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T01:58:05.251 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T01:58:05.251 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T01:58:05.251 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T01:58:05.251 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T01:58:05.251 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T01:58:05.558 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T01:58:05.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 list-attrs 2026-02-13T01:58:05.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T01:58:05.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T01:58:05.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T01:58:05.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T01:58:05.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T01:58:05.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 list-attrs 2026-02-13T01:58:05.643 INFO:tasks.workunit.client.0.trial112.stdout:_ 2026-02-13T01:58:05.643 INFO:tasks.workunit.client.0.trial112.stdout:snapset 2026-02-13T01:58:05.729 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T01:58:05.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T01:58:05.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T01:58:05.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T01:58:05.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T01:58:05.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T01:58:05.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T01:58:05.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T01:58:05.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T01:58:05.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T01:58:05.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T01:58:05.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T01:58:05.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T01:58:05.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T01:58:05.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T01:58:05.731 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T01:58:05.731 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:58:05.731 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:58:05.731 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:58:05.731 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:58:05.731 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:58:05.731 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:58:05.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:58:05.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T01:58:05.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T01:58:05.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T01:58:05.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T01:58:05.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T01:58:05.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T01:58:05.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T01:58:05.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T01:58:05.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-13T01:58:05.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T01:58:05.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T01:58:05.734 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T01:58:05.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T01:58:05.734 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T01:58:05.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T01:58:05.735 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T01:58:05.735 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T01:58:05.735 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T01:58:05.750 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:05.743+0000 7f4a6bb68a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:58:05.751 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:05.743+0000 7f4a6bb68a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:58:05.753 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:05.747+0000 7f4a6bb68a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:58:05.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T01:58:05.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T01:58:05.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T01:58:05.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T01:58:05.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T01:58:05.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:58:05.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T01:58:05.895 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T01:58:05.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:58:05.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T01:58:06.055 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:58:06.158 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:06.151+0000 7f4a6bb68a40 -1 Falling back to public interface 2026-02-13T01:58:06.755 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:06.747+0000 7f4a6bb68a40 -1 osd.0 36 log_to_monitors true 2026-02-13T01:58:07.057 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:58:07.057 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:58:07.057 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T01:58:07.057 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:58:07.058 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:58:07.058 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T01:58:07.214 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 40 up_thru 32 down_at 37 last_clean_interval [30,36) [v2:127.0.0.1:6800/2374628715,v1:127.0.0.1:6801/2374628715] [v2:127.0.0.1:6802/2374628715,v1:127.0.0.1:6803/2374628715] exists,up 64a666f3-83f2-41ab-a900-898a1e7c223a 2026-02-13T01:58:07.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T01:58:07.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T01:58:07.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T01:58:07.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T01:58:07.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T01:58:07.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T01:58:07.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T01:58:07.216 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T01:58:07.217 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T01:58:07.217 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T01:58:07.217 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T01:58:07.217 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T01:58:07.217 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T01:58:07.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T01:58:07.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T01:58:07.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T01:58:07.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T01:58:07.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T01:58:07.320 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T01:58:07.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T01:58:07.486 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:58:07.487 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T01:58:07.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T01:58:07.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:58:07.487 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T01:58:07.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691842 2026-02-13T01:58:07.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691842 2026-02-13T01:58:07.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-171798691842' 2026-02-13T01:58:07.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:58:07.561 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T01:58:07.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855369 2026-02-13T01:58:07.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855369 2026-02-13T01:58:07.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-171798691842 1-150323855369' 2026-02-13T01:58:07.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:58:07.636 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T01:58:07.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542160 2026-02-13T01:58:07.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542160 2026-02-13T01:58:07.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-171798691842 1-150323855369 2-60129542160' 2026-02-13T01:58:07.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:58:07.711 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-171798691842 2026-02-13T01:58:07.712 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:58:07.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T01:58:07.713 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-171798691842 2026-02-13T01:58:07.713 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:58:07.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691842 2026-02-13T01:58:07.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 171798691842' 2026-02-13T01:58:07.715 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 171798691842 2026-02-13T01:58:07.715 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:58:07.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691842 -lt 171798691842 2026-02-13T01:58:07.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:58:07.888 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-150323855369 2026-02-13T01:58:07.888 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:58:07.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T01:58:07.890 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-150323855369 2026-02-13T01:58:07.890 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:58:07.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855369 2026-02-13T01:58:07.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 150323855369' 2026-02-13T01:58:07.892 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 150323855369 2026-02-13T01:58:07.892 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T01:58:08.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855368 -lt 150323855369 2026-02-13T01:58:08.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:58:09.064 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T01:58:09.065 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T01:58:09.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855368 -lt 150323855369 2026-02-13T01:58:09.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:58:10.228 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T01:58:10.228 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T01:58:10.394 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855369 -lt 150323855369 2026-02-13T01:58:10.394 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:58:10.395 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542160 2026-02-13T01:58:10.395 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:58:10.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T01:58:10.397 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542160 2026-02-13T01:58:10.397 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:58:10.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542160 2026-02-13T01:58:10.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542160' 2026-02-13T01:58:10.399 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 60129542160 2026-02-13T01:58:10.399 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T01:58:10.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542161 -lt 60129542160 2026-02-13T01:58:10.566 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T01:58:10.567 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:58:10.567 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:58:10.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T01:58:10.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T01:58:10.765 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T01:58:10.765 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T01:58:10.765 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T01:58:10.765 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T01:58:10.765 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T01:58:10.765 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T01:58:10.925 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T01:58:10.926 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T01:58:10.926 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:58:10.927 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:58:11.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T01:58:11.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T01:58:11.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T01:58:11.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:647: TEST_auto_repair_bluestore_failed: rados --pool testpool get obj1 td/osd-scrub-repair/COPY 2026-02-13T01:58:11.144 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:648: TEST_auto_repair_bluestore_failed: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-13T01:58:11.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:649: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2026-02-13T01:58:11.147 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:57:54.492+0000 7faa6ded5640 10 osd.1 pg_epoch: 36 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep(periodic-regular,auto-repair)] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2026-02-13T01:58:11.147 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:57:54.492+0000 7faa6ded5640 10 osd.1 pg_epoch: 36 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep(periodic-regular,auto-repair)] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2026-02-13T01:58:11.147 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:57:54.492+0000 7faa6ded5640 15 osd.1 pg_epoch: 36 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=20'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2026-02-13T01:58:11.147 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:57:54.492+0000 7faa6ded5640 20 osd.1 pg_epoch: 36 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=20'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish: setting for deep-scrub-after-repair (5 errors. 2 errors fixed) 2026-02-13T01:58:11.147 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:57:54.492+0000 7faa6ded5640 19 osd.1 pg_epoch: 36 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=20'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2026-02-13T01:58:11.148 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:57:55.515+0000 7faa69ecd640 10 osd.1 pg_epoch: 36 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent(after-repair,check-repair)] scrubber: scrub_finish before flags: CHECK_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2026-02-13T01:58:11.148 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:57:55.515+0000 7faa69ecd640 10 osd.1 pg_epoch: 36 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent(after-repair,check-repair)] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2026-02-13T01:58:11.148 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:57:55.515+0000 7faa69ecd640 19 osd.1 pg_epoch: 36 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent(after-repair,check-repair)] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2026-02-13T01:58:11.148 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:57:55.515+0000 7faa69ecd640 10 osd.1 pg_epoch: 36 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair(after-repair)] scrubber: scrub_finish 3 error(s) still present after re-scrub 2026-02-13T01:58:11.149 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:652: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2026-02-13T01:58:11.149 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-13T01:58:11.149 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-13T01:58:11.149 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-13T01:58:11.149 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T01:58:11.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:652: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 remove 2026-02-13T01:58:11.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T01:58:11.315 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T01:58:11.315 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T01:58:11.315 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T01:58:11.315 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 remove 2026-02-13T01:58:11.315 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T01:58:11.315 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T01:58:11.315 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T01:58:11.315 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T01:58:11.315 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T01:58:11.315 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T01:58:11.316 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T01:58:11.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T01:58:11.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T01:58:11.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T01:58:11.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T01:58:11.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 remove 2026-02-13T01:58:11.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T01:58:11.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T01:58:11.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T01:58:11.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T01:58:11.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T01:58:11.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 remove 2026-02-13T01:58:11.795 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.0_head,#1:104778fc:::obj2:head#, (61) No data available 2026-02-13T01:58:11.796 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:104778fc:::obj2:head# 2026-02-13T01:58:11.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T01:58:11.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T01:58:11.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T01:58:11.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T01:58:11.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T01:58:11.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T01:58:11.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T01:58:11.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T01:58:11.908 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T01:58:11.908 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T01:58:11.908 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T01:58:11.908 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T01:58:11.908 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T01:58:11.908 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T01:58:11.908 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T01:58:11.908 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T01:58:11.908 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:58:11.908 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:58:11.908 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:58:11.908 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:58:11.909 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:58:11.909 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:58:11.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:58:11.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T01:58:11.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T01:58:11.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T01:58:11.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T01:58:11.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T01:58:11.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T01:58:11.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T01:58:11.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T01:58:11.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-13T01:58:11.910 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T01:58:11.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T01:58:11.911 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T01:58:11.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T01:58:11.911 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T01:58:11.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T01:58:11.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T01:58:11.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T01:58:11.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T01:58:11.928 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:11.919+0000 7f31c003fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:58:11.929 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:11.923+0000 7f31c003fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:58:11.930 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:11.923+0000 7f31c003fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:58:12.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T01:58:12.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T01:58:12.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T01:58:12.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T01:58:12.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T01:58:12.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:58:12.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T01:58:12.069 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T01:58:12.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:58:12.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T01:58:12.229 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:58:12.262 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:12.255+0000 7f31c003fa40 -1 Falling back to public interface 2026-02-13T01:58:12.902 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:12.895+0000 7f31c003fa40 -1 osd.1 41 log_to_monitors true 2026-02-13T01:58:13.230 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:58:13.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:58:13.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T01:58:13.231 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:58:13.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:58:13.232 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T01:58:13.390 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 45 up_thru 45 down_at 42 last_clean_interval [35,41) [v2:127.0.0.1:6810/3245293917,v1:127.0.0.1:6811/3245293917] [v2:127.0.0.1:6812/3245293917,v1:127.0.0.1:6813/3245293917] exists,up 1cfa0d8f-b9d5-4166-8842-6f71c5d0882d 2026-02-13T01:58:13.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T01:58:13.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T01:58:13.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T01:58:13.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T01:58:13.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T01:58:13.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T01:58:13.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T01:58:13.391 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T01:58:13.392 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T01:58:13.392 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T01:58:13.392 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T01:58:13.392 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T01:58:13.392 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T01:58:13.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T01:58:13.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T01:58:13.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T01:58:13.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T01:58:13.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T01:58:13.504 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T01:58:13.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T01:58:13.669 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:58:13.669 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T01:58:13.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T01:58:13.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:58:13.670 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T01:58:13.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691844 2026-02-13T01:58:13.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691844 2026-02-13T01:58:13.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-171798691844' 2026-02-13T01:58:13.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:58:13.744 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T01:58:13.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=193273528322 2026-02-13T01:58:13.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 193273528322 2026-02-13T01:58:13.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-171798691844 1-193273528322' 2026-02-13T01:58:13.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:58:13.819 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T01:58:13.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542163 2026-02-13T01:58:13.892 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542163 2026-02-13T01:58:13.892 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-171798691844 1-193273528322 2-60129542163' 2026-02-13T01:58:13.892 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:58:13.892 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-171798691844 2026-02-13T01:58:13.892 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:58:13.893 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T01:58:13.894 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-171798691844 2026-02-13T01:58:13.894 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:58:13.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691844 2026-02-13T01:58:13.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 171798691844' 2026-02-13T01:58:13.896 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 171798691844 2026-02-13T01:58:13.896 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:58:14.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691843 -lt 171798691844 2026-02-13T01:58:14.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:58:15.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T01:58:15.073 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:58:15.232 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691843 -lt 171798691844 2026-02-13T01:58:15.232 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:58:16.233 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T01:58:16.234 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:58:16.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691844 -lt 171798691844 2026-02-13T01:58:16.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:58:16.396 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-193273528322 2026-02-13T01:58:16.397 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:58:16.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T01:58:16.398 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-193273528322 2026-02-13T01:58:16.399 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:58:16.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=193273528322 2026-02-13T01:58:16.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 193273528322' 2026-02-13T01:58:16.400 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 193273528322 2026-02-13T01:58:16.401 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T01:58:16.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 193273528322 -lt 193273528322 2026-02-13T01:58:16.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:58:16.563 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542163 2026-02-13T01:58:16.563 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:58:16.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T01:58:16.565 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542163 2026-02-13T01:58:16.566 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:58:16.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542163 2026-02-13T01:58:16.568 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542163' 2026-02-13T01:58:16.568 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 60129542163 2026-02-13T01:58:16.568 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T01:58:16.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542163 -lt 60129542163 2026-02-13T01:58:16.739 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T01:58:16.739 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:58:16.739 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:58:16.927 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T01:58:16.927 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T01:58:16.928 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T01:58:16.928 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T01:58:16.928 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T01:58:16.928 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T01:58:16.928 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T01:58:16.928 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T01:58:17.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T01:58:17.086 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T01:58:17.086 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:58:17.086 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:58:17.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T01:58:17.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T01:58:17.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T01:58:17.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:653: TEST_auto_repair_bluestore_failed: repair 1.0 2026-02-13T01:58:17.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=1.0 2026-02-13T01:58:17.288 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 1.0 2026-02-13T01:58:17.288 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T01:58:17.288 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T01:58:17.288 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T01:58:17.288 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T01:58:17.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-13T01:57:55.519508+0000 2026-02-13T01:58:17.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 1.0 2026-02-13T01:58:17.600 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 1.0 on osd.1 to repair 2026-02-13T01:58:17.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 1.0 2026-02-13T01:57:55.519508+0000 2026-02-13T01:58:17.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-13T01:58:17.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T01:57:55.519508+0000 2026-02-13T01:58:17.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T01:58:17.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T01:58:17.613 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T01:58:17.613 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T01:58:17.613 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T01:58:17.613 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T01:58:17.614 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T01:58:17.614 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T01:58:17.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T01:57:55.519508+0000 '>' 2026-02-13T01:57:55.519508+0000 2026-02-13T01:58:17.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T01:58:18.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T01:58:18.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T01:58:18.779 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T01:58:18.779 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T01:58:18.779 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T01:58:18.779 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T01:58:18.779 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T01:58:18.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T01:57:55.519508+0000 '>' 2026-02-13T01:57:55.519508+0000 2026-02-13T01:58:18.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T01:58:19.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T01:58:19.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T01:58:19.944 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T01:58:19.945 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T01:58:19.945 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T01:58:19.945 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T01:58:19.945 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T01:58:20.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T01:58:17.877620+0000 '>' 2026-02-13T01:57:55.519508+0000 2026-02-13T01:58:20.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T01:58:20.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:654: TEST_auto_repair_bluestore_failed: sleep 2 2026-02-13T01:58:22.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:656: TEST_auto_repair_bluestore_failed: flush_pg_stats 2026-02-13T01:58:22.104 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T01:58:22.104 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T01:58:22.269 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T01:58:22.269 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:58:22.269 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T01:58:22.270 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T01:58:22.270 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:58:22.270 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T01:58:22.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691847 2026-02-13T01:58:22.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691847 2026-02-13T01:58:22.343 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-171798691847' 2026-02-13T01:58:22.343 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:58:22.343 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T01:58:22.416 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=193273528324 2026-02-13T01:58:22.416 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 193273528324 2026-02-13T01:58:22.416 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-171798691847 1-193273528324' 2026-02-13T01:58:22.416 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:58:22.417 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T01:58:22.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542165 2026-02-13T01:58:22.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542165 2026-02-13T01:58:22.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-171798691847 1-193273528324 2-60129542165' 2026-02-13T01:58:22.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:58:22.491 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-171798691847 2026-02-13T01:58:22.491 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:58:22.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T01:58:22.493 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-171798691847 2026-02-13T01:58:22.493 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:58:22.494 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691847 2026-02-13T01:58:22.494 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 171798691847 2026-02-13T01:58:22.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 171798691847' 2026-02-13T01:58:22.495 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:58:22.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691845 -lt 171798691847 2026-02-13T01:58:22.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:58:23.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T01:58:23.669 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:58:23.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691847 -lt 171798691847 2026-02-13T01:58:23.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:58:23.830 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-193273528324 2026-02-13T01:58:23.830 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:58:23.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T01:58:23.832 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-193273528324 2026-02-13T01:58:23.832 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:58:23.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=193273528324 2026-02-13T01:58:23.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 193273528324' 2026-02-13T01:58:23.834 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 193273528324 2026-02-13T01:58:23.834 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T01:58:24.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 193273528325 -lt 193273528324 2026-02-13T01:58:24.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:58:24.008 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542165 2026-02-13T01:58:24.008 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:58:24.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T01:58:24.010 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542165 2026-02-13T01:58:24.010 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:58:24.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542165 2026-02-13T01:58:24.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542165' 2026-02-13T01:58:24.012 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542165 2026-02-13T01:58:24.012 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T01:58:24.178 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542165 -lt 60129542165 2026-02-13T01:58:24.178 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:657: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2026-02-13T01:58:24.333 INFO:tasks.workunit.client.0.trial112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-13T01:58:24.333 INFO:tasks.workunit.client.0.trial112.stdout:1.0 9 0 0 0 0 63 0 0 10 0 10 active+clean 2026-02-13T01:58:17.877664+0000 20'10 46:153 [1,0] 1 [1,0] 1 20'10 2026-02-13T01:58:17.877620+0000 20'10 2026-02-13T01:58:17.877620+0000 0 0 periodic scrub scheduled @ 2026-02-14T01:58:17.877620+0000 9 0 2026-02-13T01:58:24.333 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:58:24.334 INFO:tasks.workunit.client.0.trial112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-13T01:58:24.334 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T01:58:24.343 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:658: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2026-02-13T01:58:24.343 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:658: TEST_auto_repair_bluestore_failed: grep -q -e '^1.0.* active+clean ' -e '^1.0.* active+clean+wait ' 2026-02-13T01:58:24.510 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T01:58:24.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:659: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2026-02-13T01:58:24.522 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:57:54.492+0000 7faa6ded5640 10 osd.1 pg_epoch: 36 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep(periodic-regular,auto-repair)] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2026-02-13T01:58:24.522 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:57:54.492+0000 7faa6ded5640 10 osd.1 pg_epoch: 36 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep(periodic-regular,auto-repair)] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2026-02-13T01:58:24.522 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:57:54.492+0000 7faa6ded5640 15 osd.1 pg_epoch: 36 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=20'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2026-02-13T01:58:24.522 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:57:54.492+0000 7faa6ded5640 20 osd.1 pg_epoch: 36 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=20'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish: setting for deep-scrub-after-repair (5 errors. 2 errors fixed) 2026-02-13T01:58:24.522 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:57:54.492+0000 7faa6ded5640 19 osd.1 pg_epoch: 36 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=20'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2026-02-13T01:58:24.522 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:57:55.515+0000 7faa69ecd640 10 osd.1 pg_epoch: 36 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent(after-repair,check-repair)] scrubber: scrub_finish before flags: CHECK_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2026-02-13T01:58:24.522 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:57:55.515+0000 7faa69ecd640 10 osd.1 pg_epoch: 36 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent(after-repair,check-repair)] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2026-02-13T01:58:24.523 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:57:55.515+0000 7faa69ecd640 19 osd.1 pg_epoch: 36 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent(after-repair,check-repair)] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2026-02-13T01:58:24.523 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:57:55.515+0000 7faa69ecd640 10 osd.1 pg_epoch: 36 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair(after-repair)] scrubber: scrub_finish 3 error(s) still present after re-scrub 2026-02-13T01:58:24.523 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:58:17.871+0000 7f319d7bc640 10 osd.1 pg_epoch: 46 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=45/46 n=10 ec=16/16 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair(must-repair,auto-repair)] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: repair. deep_scrub_on_error: 0 2026-02-13T01:58:24.523 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:58:17.871+0000 7f319d7bc640 10 osd.1 pg_epoch: 46 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=45/46 n=10 ec=16/16 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair(must-repair,auto-repair)] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2026-02-13T01:58:24.523 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:58:17.871+0000 7f319d7bc640 15 osd.1 pg_epoch: 46 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=45/46 n=9 ec=16/16 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair(must-repair,auto-repair)] scrubber: scrub_finish: 1 errors. 1 errors fixed 2026-02-13T01:58:24.523 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:58:17.871+0000 7f319d7bc640 20 osd.1 pg_epoch: 46 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=45/46 n=9 ec=16/16 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair(must-repair,auto-repair)] scrubber: scrub_finish All may be fixed 2026-02-13T01:58:24.523 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:58:17.871+0000 7f319d7bc640 19 osd.1 pg_epoch: 46 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=45/46 n=9 ec=16/16 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair(must-repair,auto-repair)] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2026-02-13T01:58:24.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-13T01:58:24.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T01:58:24.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T01:58:24.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T01:58:24.525 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T01:58:24.525 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T01:58:24.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T01:58:24.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T01:58:24.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T01:58:24.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T01:58:24.633 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T01:58:24.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T01:58:24.635 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T01:58:24.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T01:58:24.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T01:58:24.637 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T01:58:24.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T01:58:24.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T01:58:24.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T01:58:24.639 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T01:58:24.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T01:58:24.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T01:58:24.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T01:58:24.653 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T01:58:24.653 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:58:24.653 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:58:24.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T01:58:24.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T01:58:24.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T01:58:24.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-13T01:58:24.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-13T01:58:24.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-13T01:58:24.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-13T01:58:24.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T01:58:24.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T01:58:24.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T01:58:24.656 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T01:58:24.656 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T01:58:24.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T01:58:24.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T01:58:24.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T01:58:24.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T01:58:24.659 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T01:58:24.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T01:58:24.660 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T01:58:24.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T01:58:24.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T01:58:24.661 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T01:58:24.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T01:58:24.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T01:58:24.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T01:58:24.663 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T01:58:24.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T01:58:24.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T01:58:24.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T01:58:24.666 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T01:58:24.667 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:58:24.667 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:58:24.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T01:58:24.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T01:58:24.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T01:58:24.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-13T01:58:24.669 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T01:58:24.670 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:58:24.670 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:58:24.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.35207 2026-02-13T01:58:24.671 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T01:58:24.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-13T01:58:24.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T01:58:24.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-13T01:58:24.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_failed_norecov td/osd-scrub-repair 2026-02-13T01:58:24.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:663: TEST_auto_repair_bluestore_failed_norecov: local dir=td/osd-scrub-repair 2026-02-13T01:58:24.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:664: TEST_auto_repair_bluestore_failed_norecov: local poolname=testpool 2026-02-13T01:58:24.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:667: TEST_auto_repair_bluestore_failed_norecov: run_mon td/osd-scrub-repair a 2026-02-13T01:58:24.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-13T01:58:24.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-13T01:58:24.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-13T01:58:24.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-13T01:58:24.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-13T01:58:24.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-13T01:58:24.695 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-13T01:58:24.695 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:58:24.695 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:58:24.696 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:58:24.696 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:58:24.696 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:58:24.696 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:58:24.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-13T01:58:24.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-13T01:58:24.725 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-13T01:58:24.725 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T01:58:24.725 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T01:58:24.725 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-13T01:58:24.726 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-13T01:58:24.726 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T01:58:24.727 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T01:58:24.727 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T01:58:24.727 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T01:58:24.727 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:58:24.727 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:58:24.727 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T01:58:24.728 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T01:58:24.728 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get fsid 2026-02-13T01:58:24.783 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-13T01:58:24.783 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T01:58:24.783 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T01:58:24.783 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-13T01:58:24.784 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-13T01:58:24.784 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T01:58:24.784 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T01:58:24.784 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T01:58:24.785 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T01:58:24.785 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:58:24.785 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:58:24.785 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T01:58:24.785 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T01:58:24.786 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get mon_host 2026-02-13T01:58:24.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:668: TEST_auto_repair_bluestore_failed_norecov: run_mgr td/osd-scrub-repair x 2026-02-13T01:58:24.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-13T01:58:24.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-13T01:58:24.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-13T01:58:24.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-13T01:58:24.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-13T01:58:24.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-13T01:58:24.936 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-13T01:58:24.937 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:58:24.937 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:58:24.937 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:58:24.938 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:58:24.938 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:58:24.938 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:58:24.938 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T01:58:24.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T01:58:24.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:671: TEST_auto_repair_bluestore_failed_norecov: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-13T01:58:24.958 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:672: TEST_auto_repair_bluestore_failed_norecov: seq 0 2 2026-02-13T01:58:24.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:672: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2026-02-13T01:58:24.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:673: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T01:58:24.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T01:58:24.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T01:58:24.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-13T01:58:24.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T01:58:24.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T01:58:24.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T01:58:24.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T01:58:24.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T01:58:24.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T01:58:24.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T01:58:24.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T01:58:24.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T01:58:24.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T01:58:24.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T01:58:24.963 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T01:58:24.963 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:58:24.963 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:58:24.963 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:58:24.963 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:58:24.963 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:58:24.963 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:58:24.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:58:24.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T01:58:24.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T01:58:24.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T01:58:24.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T01:58:24.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T01:58:24.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T01:58:24.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T01:58:24.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T01:58:24.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T01:58:24.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T01:58:24.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-13T01:58:24.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T01:58:24.965 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T01:58:24.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5de794b3-3c03-4302-952e-878baaf16d63 2026-02-13T01:58:24.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 5de794b3-3c03-4302-952e-878baaf16d63' 2026-02-13T01:58:24.965 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 5de794b3-3c03-4302-952e-878baaf16d63 2026-02-13T01:58:24.965 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T01:58:24.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBAhY5pZsH6ORAAKc2Fu1IBxbsAlONKMmB5Vw== 2026-02-13T01:58:24.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBAhY5pZsH6ORAAKc2Fu1IBxbsAlONKMmB5Vw=="}' 2026-02-13T01:58:24.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5de794b3-3c03-4302-952e-878baaf16d63 -i td/osd-scrub-repair/0/new.json 2026-02-13T01:58:25.066 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T01:58:25.074 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-13T01:58:25.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBAhY5pZsH6ORAAKc2Fu1IBxbsAlONKMmB5Vw== --osd-uuid 5de794b3-3c03-4302-952e-878baaf16d63 2026-02-13T01:58:25.092 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:25.083+0000 7f805a776a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:58:25.094 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:25.087+0000 7f805a776a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:58:25.095 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:25.087+0000 7f805a776a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:58:25.095 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:25.087+0000 7f805a776a40 -1 bdev(0x563fc527ac00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-13T01:58:25.096 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:25.087+0000 7f805a776a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-13T01:58:25.872 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-13T01:58:25.872 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T01:58:25.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-13T01:58:25.873 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T01:58:25.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T01:58:25.980 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-13T01:58:25.980 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T01:58:25.981 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T01:58:25.981 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T01:58:25.981 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T01:58:25.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T01:58:26.006 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:25.999+0000 7fbb2b2d0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:58:26.008 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:26.003+0000 7fbb2b2d0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:58:26.010 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:26.003+0000 7fbb2b2d0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:58:26.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-13T01:58:26.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T01:58:26.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T01:58:26.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T01:58:26.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T01:58:26.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:58:26.076 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T01:58:26.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T01:58:26.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:58:26.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T01:58:26.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:58:26.518 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:26.511+0000 7fbb2b2d0a40 -1 Falling back to public interface 2026-02-13T01:58:27.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:58:27.168 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T01:58:27.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:58:27.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T01:58:27.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:58:27.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T01:58:27.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:58:27.421 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:27.415+0000 7fbb2b2d0a40 -1 osd.0 0 log_to_monitors true 2026-02-13T01:58:28.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:58:28.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:58:28.264 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T01:58:28.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T01:58:28.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:58:28.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T01:58:28.426 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1262170398,v1:127.0.0.1:6801/1262170398] [v2:127.0.0.1:6802/1262170398,v1:127.0.0.1:6803/1262170398] exists,up 5de794b3-3c03-4302-952e-878baaf16d63 2026-02-13T01:58:28.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T01:58:28.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T01:58:28.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T01:58:28.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:672: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2026-02-13T01:58:28.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:673: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T01:58:28.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T01:58:28.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T01:58:28.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-13T01:58:28.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T01:58:28.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T01:58:28.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T01:58:28.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T01:58:28.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T01:58:28.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T01:58:28.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T01:58:28.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T01:58:28.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T01:58:28.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T01:58:28.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T01:58:28.430 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T01:58:28.430 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:58:28.431 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:58:28.431 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:58:28.431 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:58:28.431 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:58:28.431 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:58:28.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:58:28.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T01:58:28.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T01:58:28.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T01:58:28.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T01:58:28.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T01:58:28.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T01:58:28.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T01:58:28.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T01:58:28.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T01:58:28.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T01:58:28.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-13T01:58:28.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T01:58:28.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T01:58:28.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6374f30a-199c-491b-9ee4-cf31d5b7f797 2026-02-13T01:58:28.433 INFO:tasks.workunit.client.0.trial112.stdout:add osd1 6374f30a-199c-491b-9ee4-cf31d5b7f797 2026-02-13T01:58:28.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 6374f30a-199c-491b-9ee4-cf31d5b7f797' 2026-02-13T01:58:28.434 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T01:58:28.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBEhY5pIhg/GhAAR0dr++iQb0W4+VlleW18NA== 2026-02-13T01:58:28.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBEhY5pIhg/GhAAR0dr++iQb0W4+VlleW18NA=="}' 2026-02-13T01:58:28.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6374f30a-199c-491b-9ee4-cf31d5b7f797 -i td/osd-scrub-repair/1/new.json 2026-02-13T01:58:28.606 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T01:58:28.615 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-13T01:58:28.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBEhY5pIhg/GhAAR0dr++iQb0W4+VlleW18NA== --osd-uuid 6374f30a-199c-491b-9ee4-cf31d5b7f797 2026-02-13T01:58:28.633 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:28.627+0000 7fd950913a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:58:28.635 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:28.627+0000 7fd950913a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:58:28.636 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:28.627+0000 7fd950913a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:58:28.637 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:28.631+0000 7fd950913a40 -1 bdev(0x5642e2f30c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-13T01:58:28.637 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:28.631+0000 7fd950913a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-13T01:58:29.315 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-13T01:58:29.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T01:58:29.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-13T01:58:29.317 INFO:tasks.workunit.client.0.trial112.stdout:adding osd1 key to auth repository 2026-02-13T01:58:29.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T01:58:29.517 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T01:58:29.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-13T01:58:29.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T01:58:29.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T01:58:29.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T01:58:29.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T01:58:29.542 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:29.535+0000 7f45b8ea5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:58:29.544 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:29.535+0000 7f45b8ea5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:58:29.545 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:29.539+0000 7f45b8ea5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:58:29.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-13T01:58:29.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T01:58:29.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T01:58:29.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T01:58:29.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T01:58:29.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:58:29.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T01:58:29.683 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T01:58:29.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:58:29.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T01:58:29.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:58:29.886 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:29.879+0000 7f45b8ea5a40 -1 Falling back to public interface 2026-02-13T01:58:30.506 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:30.499+0000 7f45b8ea5a40 -1 osd.1 0 log_to_monitors true 2026-02-13T01:58:30.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:58:30.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:58:30.842 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T01:58:30.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T01:58:30.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:58:30.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T01:58:31.016 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:58:32.017 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:58:32.017 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:58:32.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T01:58:32.018 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T01:58:32.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:58:32.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T01:58:32.174 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1720729761,v1:127.0.0.1:6811/1720729761] [v2:127.0.0.1:6812/1720729761,v1:127.0.0.1:6813/1720729761] exists,up 6374f30a-199c-491b-9ee4-cf31d5b7f797 2026-02-13T01:58:32.178 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T01:58:32.178 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T01:58:32.178 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T01:58:32.178 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:672: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2026-02-13T01:58:32.178 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:673: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T01:58:32.178 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T01:58:32.178 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T01:58:32.178 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-13T01:58:32.178 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T01:58:32.178 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-13T01:58:32.178 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T01:58:32.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T01:58:32.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T01:58:32.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T01:58:32.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-13T01:58:32.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-13T01:58:32.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T01:58:32.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T01:58:32.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T01:58:32.179 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T01:58:32.179 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:58:32.179 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:58:32.179 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:58:32.180 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:58:32.180 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:58:32.180 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:58:32.180 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:58:32.180 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T01:58:32.180 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T01:58:32.180 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T01:58:32.180 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T01:58:32.180 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T01:58:32.180 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T01:58:32.180 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T01:58:32.180 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T01:58:32.181 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T01:58:32.181 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T01:58:32.181 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-13T01:58:32.181 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-13T01:58:32.181 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T01:58:32.181 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=fa45783b-4ccd-4878-b425-06f7579902ea 2026-02-13T01:58:32.182 INFO:tasks.workunit.client.0.trial112.stdout:add osd2 fa45783b-4ccd-4878-b425-06f7579902ea 2026-02-13T01:58:32.182 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 fa45783b-4ccd-4878-b425-06f7579902ea' 2026-02-13T01:58:32.182 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T01:58:32.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBIhY5p1XtKCxAADhWIWkVPc0leXn78GHELhQ== 2026-02-13T01:58:32.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBIhY5p1XtKCxAADhWIWkVPc0leXn78GHELhQ=="}' 2026-02-13T01:58:32.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new fa45783b-4ccd-4878-b425-06f7579902ea -i td/osd-scrub-repair/2/new.json 2026-02-13T01:58:32.356 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T01:58:32.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-13T01:58:32.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBIhY5p1XtKCxAADhWIWkVPc0leXn78GHELhQ== --osd-uuid fa45783b-4ccd-4878-b425-06f7579902ea 2026-02-13T01:58:32.383 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:32.375+0000 7f80222eca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:58:32.385 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:32.379+0000 7f80222eca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:58:32.386 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:32.379+0000 7f80222eca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:58:32.386 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:32.379+0000 7f80222eca40 -1 bdev(0x56247e69cc00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-13T01:58:32.387 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:32.379+0000 7f80222eca40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-13T01:58:32.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-13T01:58:32.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T01:58:32.991 INFO:tasks.workunit.client.0.trial112.stdout:adding osd2 key to auth repository 2026-02-13T01:58:32.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-13T01:58:32.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T01:58:33.196 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-13T01:58:33.196 INFO:tasks.workunit.client.0.trial112.stdout:start osd.2 2026-02-13T01:58:33.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T01:58:33.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T01:58:33.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T01:58:33.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T01:58:33.222 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:33.215+0000 7f7eac627a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:58:33.224 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:33.219+0000 7f7eac627a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:58:33.225 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:33.219+0000 7f7eac627a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:58:33.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-13T01:58:33.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T01:58:33.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T01:58:33.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T01:58:33.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T01:58:33.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:58:33.356 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T01:58:33.357 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T01:58:33.357 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:58:33.357 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T01:58:33.515 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:58:33.602 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:33.595+0000 7f7eac627a40 -1 Falling back to public interface 2026-02-13T01:58:34.214 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:34.207+0000 7f7eac627a40 -1 osd.2 0 log_to_monitors true 2026-02-13T01:58:34.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:58:34.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:58:34.517 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T01:58:34.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T01:58:34.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:58:34.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T01:58:34.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:58:35.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:58:35.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:58:35.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T01:58:35.686 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T01:58:35.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:58:35.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T01:58:35.842 INFO:tasks.workunit.client.0.trial112.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3133793372,v1:127.0.0.1:6819/3133793372] [v2:127.0.0.1:6820/3133793372,v1:127.0.0.1:6821/3133793372] exists,up fa45783b-4ccd-4878-b425-06f7579902ea 2026-02-13T01:58:35.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T01:58:35.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T01:58:35.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T01:58:35.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_auto_repair_bluestore_failed_norecov: create_pool testpool 1 1 2026-02-13T01:58:35.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create testpool 1 1 2026-02-13T01:58:36.087 INFO:tasks.workunit.client.0.trial112.stderr:pool 'testpool' created 2026-02-13T01:58:36.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T01:58:37.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_auto_repair_bluestore_failed_norecov: ceph osd pool set testpool size 2 2026-02-13T01:58:37.302 INFO:tasks.workunit.client.0.trial112.stderr:set pool 1 size to 2 2026-02-13T01:58:37.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:678: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2026-02-13T01:58:37.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T01:58:37.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T01:58:37.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T01:58:37.315 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T01:58:37.316 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T01:58:37.316 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T01:58:37.316 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T01:58:37.316 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T01:58:37.316 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T01:58:37.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T01:58:37.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T01:58:37.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T01:58:37.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T01:58:37.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T01:58:37.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T01:58:37.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T01:58:37.589 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:58:37.589 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T01:58:37.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T01:58:37.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:58:37.590 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T01:58:37.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-13T01:58:37.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-13T01:58:37.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-13T01:58:37.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:58:37.665 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T01:58:37.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672963 2026-02-13T01:58:37.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672963 2026-02-13T01:58:37.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-42949672963' 2026-02-13T01:58:37.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:58:37.737 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T01:58:37.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542146 2026-02-13T01:58:37.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542146 2026-02-13T01:58:37.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-42949672963 2-60129542146' 2026-02-13T01:58:37.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:58:37.809 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-13T01:58:37.809 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:58:37.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T01:58:37.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-13T01:58:37.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:58:37.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-13T01:58:37.813 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836483 2026-02-13T01:58:37.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-13T01:58:37.814 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:58:37.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-13T01:58:37.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:58:38.980 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T01:58:38.980 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:58:39.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-13T01:58:39.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:58:40.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T01:58:40.147 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:58:40.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836483 2026-02-13T01:58:40.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:58:40.304 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672963 2026-02-13T01:58:40.304 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:58:40.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T01:58:40.306 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672963 2026-02-13T01:58:40.306 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:58:40.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672963 2026-02-13T01:58:40.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2026-02-13T01:58:40.307 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 42949672963 2026-02-13T01:58:40.308 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T01:58:40.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672963 -lt 42949672963 2026-02-13T01:58:40.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:58:40.478 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542146 2026-02-13T01:58:40.479 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:58:40.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T01:58:40.481 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542146 2026-02-13T01:58:40.481 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:58:40.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542146 2026-02-13T01:58:40.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542146' 2026-02-13T01:58:40.483 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542146 2026-02-13T01:58:40.483 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T01:58:40.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542146 -lt 60129542146 2026-02-13T01:58:40.655 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T01:58:40.655 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:58:40.655 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:58:40.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T01:58:40.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T01:58:40.855 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T01:58:40.855 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T01:58:40.855 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T01:58:40.855 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T01:58:40.856 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T01:58:40.856 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T01:58:41.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T01:58:41.014 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T01:58:41.014 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:58:41.014 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:58:41.210 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T01:58:41.210 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T01:58:41.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T01:58:41.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:681: TEST_auto_repair_bluestore_failed_norecov: local payload=ABCDEF 2026-02-13T01:58:41.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:682: TEST_auto_repair_bluestore_failed_norecov: echo ABCDEF 2026-02-13T01:58:41.211 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:683: TEST_auto_repair_bluestore_failed_norecov: seq 1 10 2026-02-13T01:58:41.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:683: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2026-02-13T01:58:41.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2026-02-13T01:58:41.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:683: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2026-02-13T01:58:41.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2026-02-13T01:58:41.258 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:683: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2026-02-13T01:58:41.258 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2026-02-13T01:58:41.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:683: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2026-02-13T01:58:41.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2026-02-13T01:58:41.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:683: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2026-02-13T01:58:41.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2026-02-13T01:58:41.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:683: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2026-02-13T01:58:41.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2026-02-13T01:58:41.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:683: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2026-02-13T01:58:41.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2026-02-13T01:58:41.385 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:683: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2026-02-13T01:58:41.385 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2026-02-13T01:58:41.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:683: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2026-02-13T01:58:41.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2026-02-13T01:58:41.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:683: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2026-02-13T01:58:41.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2026-02-13T01:58:41.463 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2026-02-13T01:58:41.463 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-13T01:58:41.463 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-13T01:58:41.464 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2026-02-13T01:58:41.465 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-13T01:58:41.465 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-13T01:58:41.465 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-13T01:58:41.465 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T01:58:41.627 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-13T01:58:41.627 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2026-02-13T01:58:41.627 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-13T01:58:41.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2026-02-13T01:58:41.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T01:58:41.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T01:58:41.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T01:58:41.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T01:58:41.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2026-02-13T01:58:41.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T01:58:41.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T01:58:41.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T01:58:41.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T01:58:41.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T01:58:41.790 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T01:58:41.790 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T01:58:41.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T01:58:41.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T01:58:41.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T01:58:42.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T01:58:42.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2026-02-13T01:58:42.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T01:58:42.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T01:58:42.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T01:58:42.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T01:58:42.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T01:58:42.099 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2026-02-13T01:58:42.292 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:ff7b1f36:::obj1:head# 2026-02-13T01:58:42.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T01:58:42.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T01:58:42.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T01:58:42.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T01:58:42.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T01:58:42.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T01:58:42.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T01:58:42.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T01:58:42.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T01:58:42.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T01:58:42.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T01:58:42.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T01:58:42.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T01:58:42.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T01:58:42.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T01:58:42.393 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T01:58:42.393 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:58:42.393 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:58:42.393 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:58:42.393 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:58:42.393 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:58:42.393 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:58:42.393 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:58:42.393 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T01:58:42.393 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T01:58:42.393 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T01:58:42.393 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T01:58:42.393 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T01:58:42.394 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T01:58:42.394 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T01:58:42.394 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T01:58:42.394 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-13T01:58:42.394 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T01:58:42.395 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T01:58:42.395 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T01:58:42.395 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T01:58:42.395 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T01:58:42.395 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T01:58:42.395 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T01:58:42.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T01:58:42.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T01:58:42.415 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:42.407+0000 7fd44be13a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:58:42.416 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:42.407+0000 7fd44be13a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:58:42.417 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:42.411+0000 7fd44be13a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:58:42.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T01:58:42.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T01:58:42.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T01:58:42.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T01:58:42.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T01:58:42.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:58:42.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T01:58:42.555 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T01:58:42.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:58:42.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T01:58:42.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:58:42.782 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:42.775+0000 7fd44be13a40 -1 Falling back to public interface 2026-02-13T01:58:43.395 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:43.387+0000 7fd44be13a40 -1 osd.0 19 log_to_monitors true 2026-02-13T01:58:43.718 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:58:43.718 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:58:43.718 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T01:58:43.718 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:58:43.718 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:58:43.718 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T01:58:43.883 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 23 up_thru 0 down_at 20 last_clean_interval [5,19) [v2:127.0.0.1:6800/711484853,v1:127.0.0.1:6801/711484853] [v2:127.0.0.1:6802/711484853,v1:127.0.0.1:6803/711484853] exists,up 5de794b3-3c03-4302-952e-878baaf16d63 2026-02-13T01:58:43.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T01:58:43.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T01:58:43.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T01:58:43.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T01:58:43.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T01:58:43.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T01:58:43.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T01:58:43.883 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T01:58:43.884 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T01:58:43.885 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T01:58:43.885 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T01:58:43.885 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T01:58:43.885 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T01:58:43.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T01:58:43.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T01:58:43.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T01:58:43.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T01:58:43.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T01:58:43.993 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T01:58:44.152 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T01:58:44.153 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:58:44.153 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T01:58:44.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T01:58:44.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:58:44.153 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T01:58:44.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247810 2026-02-13T01:58:44.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247810 2026-02-13T01:58:44.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247810' 2026-02-13T01:58:44.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:58:44.227 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T01:58:44.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672965 2026-02-13T01:58:44.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672965 2026-02-13T01:58:44.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247810 1-42949672965' 2026-02-13T01:58:44.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:58:44.301 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T01:58:44.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542148 2026-02-13T01:58:44.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542148 2026-02-13T01:58:44.376 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247810 1-42949672965 2-60129542148' 2026-02-13T01:58:44.376 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:58:44.376 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-98784247810 2026-02-13T01:58:44.376 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:58:44.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T01:58:44.378 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-98784247810 2026-02-13T01:58:44.378 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:58:44.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247810 2026-02-13T01:58:44.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 98784247810' 2026-02-13T01:58:44.380 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 98784247810 2026-02-13T01:58:44.380 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:58:44.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 98784247810 2026-02-13T01:58:44.547 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:58:45.548 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T01:58:45.548 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:58:45.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247810 -lt 98784247810 2026-02-13T01:58:45.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:58:45.711 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672965 2026-02-13T01:58:45.711 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:58:45.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T01:58:45.713 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672965 2026-02-13T01:58:45.714 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:58:45.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672965 2026-02-13T01:58:45.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2026-02-13T01:58:45.715 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 42949672965 2026-02-13T01:58:45.716 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T01:58:45.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672965 -lt 42949672965 2026-02-13T01:58:45.886 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:58:45.886 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542148 2026-02-13T01:58:45.886 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:58:45.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T01:58:45.888 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542148 2026-02-13T01:58:45.889 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:58:45.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542148 2026-02-13T01:58:45.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542148' 2026-02-13T01:58:45.890 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 60129542148 2026-02-13T01:58:45.891 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T01:58:46.065 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542148 -lt 60129542148 2026-02-13T01:58:46.066 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T01:58:46.066 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:58:46.067 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:58:46.266 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T01:58:46.266 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T01:58:46.267 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T01:58:46.267 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T01:58:46.267 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T01:58:46.268 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T01:58:46.268 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T01:58:46.268 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T01:58:46.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T01:58:46.427 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T01:58:46.427 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:58:46.427 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:58:46.621 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T01:58:46.621 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T01:58:46.621 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T01:58:46.621 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:692: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2026-02-13T01:58:46.621 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-13T01:58:46.621 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-13T01:58:46.622 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-13T01:58:46.622 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T01:58:46.779 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:692: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj1 rm-attr _ 2026-02-13T01:58:46.779 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T01:58:46.779 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T01:58:46.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T01:58:46.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T01:58:46.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj1 rm-attr _ 2026-02-13T01:58:46.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T01:58:46.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T01:58:46.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T01:58:46.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T01:58:46.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T01:58:46.780 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T01:58:46.780 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T01:58:46.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T01:58:46.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T01:58:46.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T01:58:47.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T01:58:47.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 rm-attr _ 2026-02-13T01:58:47.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T01:58:47.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T01:58:47.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T01:58:47.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T01:58:47.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T01:58:47.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 rm-attr _ 2026-02-13T01:58:47.403 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T01:58:47.403 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T01:58:47.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T01:58:47.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T01:58:47.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T01:58:47.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T01:58:47.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T01:58:47.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T01:58:47.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T01:58:47.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T01:58:47.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T01:58:47.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T01:58:47.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T01:58:47.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T01:58:47.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T01:58:47.405 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T01:58:47.405 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:58:47.405 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:58:47.405 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:58:47.405 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:58:47.405 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:58:47.405 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:58:47.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:58:47.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T01:58:47.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T01:58:47.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T01:58:47.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T01:58:47.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T01:58:47.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T01:58:47.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T01:58:47.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T01:58:47.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-13T01:58:47.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T01:58:47.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T01:58:47.407 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T01:58:47.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T01:58:47.408 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T01:58:47.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T01:58:47.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T01:58:47.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T01:58:47.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T01:58:47.424 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:47.419+0000 7f50beeeba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:58:47.426 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:47.419+0000 7f50beeeba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:58:47.427 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:47.419+0000 7f50beeeba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:58:47.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T01:58:47.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T01:58:47.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T01:58:47.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T01:58:47.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T01:58:47.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:58:47.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T01:58:47.571 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T01:58:47.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:58:47.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T01:58:47.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:58:47.902 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:47.895+0000 7f50beeeba40 -1 Falling back to public interface 2026-02-13T01:58:48.511 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:48.503+0000 7f50beeeba40 -1 osd.1 25 log_to_monitors true 2026-02-13T01:58:48.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:58:48.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:58:48.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T01:58:48.732 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:58:48.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:58:48.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T01:58:48.900 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 29 up_thru 29 down_at 25 last_clean_interval [10,25) [v2:127.0.0.1:6810/1343623056,v1:127.0.0.1:6811/1343623056] [v2:127.0.0.1:6812/1343623056,v1:127.0.0.1:6813/1343623056] exists,up 6374f30a-199c-491b-9ee4-cf31d5b7f797 2026-02-13T01:58:48.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T01:58:48.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T01:58:48.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T01:58:48.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T01:58:48.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T01:58:48.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T01:58:48.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T01:58:48.900 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T01:58:48.901 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T01:58:48.901 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T01:58:48.901 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T01:58:48.902 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T01:58:48.902 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T01:58:49.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T01:58:49.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T01:58:49.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T01:58:49.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T01:58:49.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T01:58:49.011 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T01:58:49.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T01:58:49.174 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:58:49.174 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T01:58:49.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T01:58:49.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:58:49.175 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T01:58:49.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247812 2026-02-13T01:58:49.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247812 2026-02-13T01:58:49.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247812' 2026-02-13T01:58:49.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:58:49.248 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T01:58:49.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=124554051586 2026-02-13T01:58:49.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 124554051586 2026-02-13T01:58:49.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247812 1-124554051586' 2026-02-13T01:58:49.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:58:49.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T01:58:49.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542150 2026-02-13T01:58:49.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542150 2026-02-13T01:58:49.397 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247812 1-124554051586 2-60129542150' 2026-02-13T01:58:49.397 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:58:49.397 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-98784247812 2026-02-13T01:58:49.397 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:58:49.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T01:58:49.399 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-98784247812 2026-02-13T01:58:49.399 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:58:49.401 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247812 2026-02-13T01:58:49.401 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 98784247812' 2026-02-13T01:58:49.401 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 98784247812 2026-02-13T01:58:49.401 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:58:49.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247811 -lt 98784247812 2026-02-13T01:58:49.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:58:50.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T01:58:50.566 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:58:50.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247811 -lt 98784247812 2026-02-13T01:58:50.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:58:51.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T01:58:51.726 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:58:51.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247812 -lt 98784247812 2026-02-13T01:58:51.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:58:51.892 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-124554051586 2026-02-13T01:58:51.892 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:58:51.893 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T01:58:51.894 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-124554051586 2026-02-13T01:58:51.894 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:58:51.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=124554051586 2026-02-13T01:58:51.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 124554051586' 2026-02-13T01:58:51.896 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 124554051586 2026-02-13T01:58:51.896 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T01:58:52.057 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 124554051586 -lt 124554051586 2026-02-13T01:58:52.057 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:58:52.058 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542150 2026-02-13T01:58:52.058 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:58:52.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T01:58:52.060 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542150 2026-02-13T01:58:52.060 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:58:52.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542150 2026-02-13T01:58:52.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542150' 2026-02-13T01:58:52.061 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 60129542150 2026-02-13T01:58:52.062 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T01:58:52.228 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542151 -lt 60129542150 2026-02-13T01:58:52.228 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T01:58:52.229 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:58:52.229 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:58:52.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T01:58:52.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T01:58:52.426 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T01:58:52.426 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T01:58:52.426 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T01:58:52.426 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T01:58:52.426 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T01:58:52.426 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T01:58:52.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T01:58:52.583 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T01:58:52.584 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:58:52.584 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:58:52.775 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T01:58:52.775 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T01:58:52.775 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T01:58:52.776 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2026-02-13T01:58:52.776 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-13T01:58:52.777 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-13T01:58:52.777 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2026-02-13T01:58:52.777 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-13T01:58:52.777 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-13T01:58:52.777 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-13T01:58:52.777 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T01:58:52.942 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-13T01:58:52.942 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2026-02-13T01:58:52.943 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-13T01:58:53.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2026-02-13T01:58:53.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T01:58:53.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T01:58:53.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T01:58:53.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T01:58:53.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2026-02-13T01:58:53.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T01:58:53.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T01:58:53.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T01:58:53.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T01:58:53.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T01:58:53.103 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T01:58:53.103 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T01:58:53.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T01:58:53.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T01:58:53.104 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T01:58:53.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T01:58:53.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2026-02-13T01:58:53.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T01:58:53.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T01:58:53.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T01:58:53.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T01:58:53.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T01:58:53.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2026-02-13T01:58:53.564 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:104778fc:::obj2:head# 2026-02-13T01:58:53.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T01:58:53.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T01:58:53.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T01:58:53.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T01:58:53.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T01:58:53.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T01:58:53.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T01:58:53.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T01:58:53.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T01:58:53.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T01:58:53.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T01:58:53.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T01:58:53.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T01:58:53.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T01:58:53.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T01:58:53.667 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T01:58:53.667 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:58:53.668 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:58:53.668 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:58:53.668 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:58:53.668 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:58:53.668 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:58:53.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:58:53.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T01:58:53.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T01:58:53.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T01:58:53.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T01:58:53.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T01:58:53.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T01:58:53.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T01:58:53.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T01:58:53.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-13T01:58:53.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T01:58:53.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T01:58:53.671 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T01:58:53.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T01:58:53.671 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T01:58:53.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T01:58:53.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T01:58:53.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T01:58:53.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T01:58:53.686 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:53.679+0000 7f481cff8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:58:53.687 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:53.679+0000 7f481cff8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:58:53.688 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:53.679+0000 7f481cff8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:58:53.836 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T01:58:53.836 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T01:58:53.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T01:58:53.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T01:58:53.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T01:58:53.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:58:53.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T01:58:53.837 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T01:58:53.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:58:53.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T01:58:53.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:58:54.054 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:54.047+0000 7f481cff8a40 -1 Falling back to public interface 2026-02-13T01:58:54.618 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:54.611+0000 7f481cff8a40 -1 osd.0 31 log_to_monitors true 2026-02-13T01:58:54.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:58:54.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:58:54.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T01:58:54.999 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:58:55.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:58:55.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T01:58:55.155 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 35 up_thru 26 down_at 32 last_clean_interval [23,31) [v2:127.0.0.1:6800/898819727,v1:127.0.0.1:6801/898819727] [v2:127.0.0.1:6802/898819727,v1:127.0.0.1:6803/898819727] exists,up 5de794b3-3c03-4302-952e-878baaf16d63 2026-02-13T01:58:55.155 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T01:58:55.155 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T01:58:55.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T01:58:55.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T01:58:55.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T01:58:55.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T01:58:55.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T01:58:55.156 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T01:58:55.157 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T01:58:55.157 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T01:58:55.157 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T01:58:55.157 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T01:58:55.157 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T01:58:55.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T01:58:55.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T01:58:55.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T01:58:55.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T01:58:55.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T01:58:55.265 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T01:58:55.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T01:58:55.427 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:58:55.427 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T01:58:55.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T01:58:55.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:58:55.428 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T01:58:55.502 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855362 2026-02-13T01:58:55.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855362 2026-02-13T01:58:55.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-150323855362' 2026-02-13T01:58:55.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:58:55.503 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T01:58:55.576 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=124554051588 2026-02-13T01:58:55.576 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 124554051588 2026-02-13T01:58:55.576 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-150323855362 1-124554051588' 2026-02-13T01:58:55.576 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:58:55.576 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T01:58:55.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542153 2026-02-13T01:58:55.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542153 2026-02-13T01:58:55.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-150323855362 1-124554051588 2-60129542153' 2026-02-13T01:58:55.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:58:55.652 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-150323855362 2026-02-13T01:58:55.652 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:58:55.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T01:58:55.653 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-150323855362 2026-02-13T01:58:55.654 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:58:55.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855362 2026-02-13T01:58:55.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2026-02-13T01:58:55.655 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 150323855362 2026-02-13T01:58:55.656 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:58:55.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 150323855362 2026-02-13T01:58:55.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:58:56.827 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T01:58:56.828 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:58:56.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 150323855362 2026-02-13T01:58:56.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:58:57.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T01:58:57.993 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:58:58.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855362 -lt 150323855362 2026-02-13T01:58:58.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:58:58.162 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-124554051588 2026-02-13T01:58:58.162 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:58:58.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T01:58:58.163 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-124554051588 2026-02-13T01:58:58.164 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:58:58.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=124554051588 2026-02-13T01:58:58.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 124554051588' 2026-02-13T01:58:58.165 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 124554051588 2026-02-13T01:58:58.166 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T01:58:58.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 124554051588 -lt 124554051588 2026-02-13T01:58:58.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:58:58.327 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542153 2026-02-13T01:58:58.327 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:58:58.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T01:58:58.329 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542153 2026-02-13T01:58:58.330 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:58:58.331 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542153 2026-02-13T01:58:58.331 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542153' 2026-02-13T01:58:58.331 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 60129542153 2026-02-13T01:58:58.332 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T01:58:58.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542153 -lt 60129542153 2026-02-13T01:58:58.504 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T01:58:58.504 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:58:58.504 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:58:58.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T01:58:58.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T01:58:58.704 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T01:58:58.704 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T01:58:58.704 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T01:58:58.704 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T01:58:58.704 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T01:58:58.704 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T01:58:58.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T01:58:58.863 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T01:58:58.863 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:58:58.863 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:58:59.057 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T01:58:59.057 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T01:58:59.057 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T01:58:59.058 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:695: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2026-02-13T01:58:59.058 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-13T01:58:59.059 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-13T01:58:59.059 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-13T01:58:59.059 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T01:58:59.222 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:695: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2026-02-13T01:58:59.222 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T01:58:59.222 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T01:58:59.223 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T01:58:59.223 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T01:58:59.223 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2026-02-13T01:58:59.223 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T01:58:59.223 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T01:58:59.223 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T01:58:59.223 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T01:58:59.223 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T01:58:59.223 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T01:58:59.223 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T01:58:59.223 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T01:58:59.223 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T01:58:59.223 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T01:58:59.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T01:58:59.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2026-02-13T01:58:59.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T01:58:59.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T01:58:59.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T01:58:59.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T01:58:59.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T01:58:59.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2026-02-13T01:58:59.736 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.0_head,#1:ff7b1f36:::obj1:head#, (61) No data available 2026-02-13T01:58:59.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T01:58:59.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T01:58:59.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T01:58:59.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T01:58:59.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T01:58:59.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T01:58:59.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T01:58:59.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T01:58:59.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T01:58:59.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T01:58:59.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T01:58:59.835 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T01:58:59.835 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T01:58:59.835 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T01:58:59.835 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T01:58:59.835 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T01:58:59.835 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:58:59.835 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:58:59.835 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:58:59.835 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:58:59.835 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:58:59.837 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:58:59.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:58:59.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T01:58:59.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T01:58:59.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T01:58:59.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T01:58:59.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T01:58:59.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T01:58:59.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T01:58:59.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T01:58:59.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-13T01:58:59.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T01:58:59.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T01:58:59.838 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T01:58:59.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T01:58:59.839 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T01:58:59.839 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T01:58:59.839 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T01:58:59.839 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T01:58:59.839 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T01:58:59.856 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:59.851+0000 7fd844120a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:58:59.857 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:59.851+0000 7fd844120a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:58:59.858 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:58:59.851+0000 7fd844120a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:59:00.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T01:59:00.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T01:59:00.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T01:59:00.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T01:59:00.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T01:59:00.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:59:00.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T01:59:00.001 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T01:59:00.001 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:59:00.001 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T01:59:00.160 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:59:00.278 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:59:00.271+0000 7fd844120a40 -1 Falling back to public interface 2026-02-13T01:59:00.902 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:59:00.895+0000 7fd844120a40 -1 osd.1 36 log_to_monitors true 2026-02-13T01:59:01.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:59:01.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:59:01.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T01:59:01.162 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:59:01.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:59:01.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T01:59:01.321 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 40 up_thru 40 down_at 37 last_clean_interval [29,36) [v2:127.0.0.1:6810/4251004974,v1:127.0.0.1:6811/4251004974] [v2:127.0.0.1:6812/4251004974,v1:127.0.0.1:6813/4251004974] exists,up 6374f30a-199c-491b-9ee4-cf31d5b7f797 2026-02-13T01:59:01.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T01:59:01.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T01:59:01.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T01:59:01.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T01:59:01.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T01:59:01.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T01:59:01.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T01:59:01.322 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T01:59:01.322 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T01:59:01.322 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T01:59:01.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T01:59:01.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T01:59:01.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T01:59:01.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T01:59:01.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T01:59:01.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T01:59:01.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T01:59:01.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T01:59:01.425 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T01:59:01.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T01:59:01.590 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:59:01.590 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T01:59:01.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T01:59:01.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:59:01.591 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T01:59:01.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855364 2026-02-13T01:59:01.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855364 2026-02-13T01:59:01.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-150323855364' 2026-02-13T01:59:01.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:59:01.664 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T01:59:01.735 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691842 2026-02-13T01:59:01.735 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691842 2026-02-13T01:59:01.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-150323855364 1-171798691842' 2026-02-13T01:59:01.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:59:01.736 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T01:59:01.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542155 2026-02-13T01:59:01.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542155 2026-02-13T01:59:01.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-150323855364 1-171798691842 2-60129542155' 2026-02-13T01:59:01.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:59:01.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-150323855364 2026-02-13T01:59:01.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:59:01.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T01:59:01.813 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-150323855364 2026-02-13T01:59:01.813 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:59:01.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855364 2026-02-13T01:59:01.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 150323855364' 2026-02-13T01:59:01.815 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 150323855364 2026-02-13T01:59:01.815 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:59:01.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855363 -lt 150323855364 2026-02-13T01:59:01.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:59:02.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T01:59:02.985 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:59:03.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855363 -lt 150323855364 2026-02-13T01:59:03.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:59:04.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T01:59:04.148 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:59:04.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855364 -lt 150323855364 2026-02-13T01:59:04.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:59:04.313 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-171798691842 2026-02-13T01:59:04.313 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:59:04.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T01:59:04.315 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-171798691842 2026-02-13T01:59:04.315 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:59:04.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691842 2026-02-13T01:59:04.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 171798691842' 2026-02-13T01:59:04.317 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 171798691842 2026-02-13T01:59:04.317 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T01:59:04.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691842 -lt 171798691842 2026-02-13T01:59:04.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:59:04.482 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542155 2026-02-13T01:59:04.482 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:59:04.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T01:59:04.484 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542155 2026-02-13T01:59:04.484 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:59:04.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542155 2026-02-13T01:59:04.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542155' 2026-02-13T01:59:04.485 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 60129542155 2026-02-13T01:59:04.485 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T01:59:04.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542155 -lt 60129542155 2026-02-13T01:59:04.653 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T01:59:04.653 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:59:04.654 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:59:04.847 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T01:59:04.847 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T01:59:04.847 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T01:59:04.847 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T01:59:04.847 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T01:59:04.847 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T01:59:04.847 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T01:59:04.847 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T01:59:05.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T01:59:05.004 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T01:59:05.004 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:59:05.004 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:59:05.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T01:59:05.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T01:59:05.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T01:59:05.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:696: TEST_auto_repair_bluestore_failed_norecov: ceph tell 'osd.*' config set osd_scrub_auto_repair true 2026-02-13T01:59:05.262 INFO:tasks.workunit.client.0.trial112.stdout:osd.0: { 2026-02-13T01:59:05.263 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-13T01:59:05.263 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T01:59:05.270 INFO:tasks.workunit.client.0.trial112.stdout:osd.1: { 2026-02-13T01:59:05.270 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-13T01:59:05.270 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T01:59:05.277 INFO:tasks.workunit.client.0.trial112.stdout:osd.2: { 2026-02-13T01:59:05.277 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2026-02-13T01:59:05.277 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T01:59:05.286 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:698: TEST_auto_repair_bluestore_failed_norecov: get_pg testpool obj1 2026-02-13T01:59:05.286 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2026-02-13T01:59:05.286 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=obj1 2026-02-13T01:59:05.286 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool obj1 2026-02-13T01:59:05.286 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T01:59:05.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:698: TEST_auto_repair_bluestore_failed_norecov: local pgid=1.0 2026-02-13T01:59:05.447 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:699: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool obj1 2026-02-13T01:59:05.447 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-13T01:59:05.448 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-13T01:59:05.448 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2026-02-13T01:59:05.448 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T01:59:05.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:699: TEST_auto_repair_bluestore_failed_norecov: local primary=1 2026-02-13T01:59:05.607 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:700: TEST_auto_repair_bluestore_failed_norecov: get_last_scrub_stamp 1.0 2026-02-13T01:59:05.608 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T01:59:05.608 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T01:59:05.608 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T01:59:05.608 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T01:59:05.770 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:700: TEST_auto_repair_bluestore_failed_norecov: local last_scrub_stamp=2026-02-13T01:58:36.082993+0000 2026-02-13T01:59:05.770 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:701: TEST_auto_repair_bluestore_failed_norecov: ceph tell 1.0 schedule-deep-scrub 2026-02-13T01:59:05.836 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T01:59:05.836 INFO:tasks.workunit.client.0.trial112.stdout: "deep": true, 2026-02-13T01:59:05.836 INFO:tasks.workunit.client.0.trial112.stdout: "must": false, 2026-02-13T01:59:05.836 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "2026-02-01T21:10:55.835335+0000" 2026-02-13T01:59:05.837 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T01:59:05.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:704: TEST_auto_repair_bluestore_failed_norecov: wait_for_scrub 1.0 2026-02-13T01:58:36.082993+0000 2026-02-13T01:59:05.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-13T01:59:05.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T01:58:36.082993+0000 2026-02-13T01:59:05.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T01:59:05.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T01:59:05.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T01:59:05.844 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T01:59:05.844 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T01:59:05.844 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T01:59:05.845 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T01:59:05.845 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T01:59:06.006 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T01:58:36.082993+0000 '>' 2026-02-13T01:58:36.082993+0000 2026-02-13T01:59:06.006 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T01:59:07.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T01:59:07.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T01:59:07.009 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T01:59:07.009 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T01:59:07.009 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T01:59:07.009 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T01:59:07.010 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T01:59:07.170 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T01:58:36.082993+0000 '>' 2026-02-13T01:58:36.082993+0000 2026-02-13T01:59:07.170 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T01:59:08.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T01:59:08.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T01:59:08.172 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T01:59:08.172 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T01:59:08.172 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T01:59:08.173 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T01:59:08.173 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T01:59:08.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T01:58:36.082993+0000 '>' 2026-02-13T01:58:36.082993+0000 2026-02-13T01:59:08.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T01:59:09.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T01:59:09.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T01:59:09.336 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T01:59:09.336 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T01:59:09.336 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T01:59:09.336 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T01:59:09.336 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T01:59:09.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T01:58:36.082993+0000 '>' 2026-02-13T01:58:36.082993+0000 2026-02-13T01:59:09.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T01:59:10.498 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T01:59:10.498 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T01:59:10.499 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T01:59:10.499 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T01:59:10.499 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T01:59:10.499 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T01:59:10.499 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T01:59:10.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T01:58:36.082993+0000 '>' 2026-02-13T01:58:36.082993+0000 2026-02-13T01:59:10.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T01:59:11.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T01:59:11.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T01:59:11.656 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T01:59:11.656 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T01:59:11.656 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T01:59:11.656 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T01:59:11.657 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T01:59:11.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T01:59:10.722825+0000 '>' 2026-02-13T01:58:36.082993+0000 2026-02-13T01:59:11.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T01:59:11.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:705: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2026-02-13T01:59:11.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T01:59:11.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T01:59:11.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T01:59:11.814 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T01:59:11.815 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T01:59:11.815 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T01:59:11.815 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T01:59:11.815 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T01:59:11.815 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T01:59:11.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T01:59:11.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T01:59:11.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T01:59:11.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T01:59:11.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T01:59:11.923 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T01:59:12.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T01:59:12.080 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:59:12.080 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T01:59:12.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T01:59:12.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:59:12.081 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T01:59:12.154 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855367 2026-02-13T01:59:12.154 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855367 2026-02-13T01:59:12.154 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-150323855367' 2026-02-13T01:59:12.154 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:59:12.155 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T01:59:12.230 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691845 2026-02-13T01:59:12.230 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691845 2026-02-13T01:59:12.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-150323855367 1-171798691845' 2026-02-13T01:59:12.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:59:12.231 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T01:59:12.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542158 2026-02-13T01:59:12.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542158 2026-02-13T01:59:12.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-150323855367 1-171798691845 2-60129542158' 2026-02-13T01:59:12.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:59:12.304 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-150323855367 2026-02-13T01:59:12.304 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:59:12.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T01:59:12.305 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-150323855367 2026-02-13T01:59:12.305 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:59:12.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855367 2026-02-13T01:59:12.307 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 150323855367 2026-02-13T01:59:12.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 150323855367' 2026-02-13T01:59:12.308 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:59:12.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855366 -lt 150323855367 2026-02-13T01:59:12.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:59:13.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T01:59:13.480 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:59:13.646 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855367 -lt 150323855367 2026-02-13T01:59:13.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:59:13.647 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-171798691845 2026-02-13T01:59:13.647 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:59:13.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T01:59:13.649 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-171798691845 2026-02-13T01:59:13.650 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:59:13.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691845 2026-02-13T01:59:13.651 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 171798691845 2026-02-13T01:59:13.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 171798691845' 2026-02-13T01:59:13.652 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T01:59:13.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691845 -lt 171798691845 2026-02-13T01:59:13.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:59:13.820 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542158 2026-02-13T01:59:13.820 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:59:13.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T01:59:13.822 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542158 2026-02-13T01:59:13.822 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:59:13.823 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542158 2026-02-13T01:59:13.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542158' 2026-02-13T01:59:13.824 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542158 2026-02-13T01:59:13.824 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T01:59:13.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542158 -lt 60129542158 2026-02-13T01:59:13.998 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T01:59:13.998 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:59:13.998 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:59:14.196 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T01:59:14.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T01:59:14.198 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T01:59:14.198 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T01:59:14.198 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T01:59:14.198 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T01:59:14.198 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T01:59:14.198 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T01:59:14.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T01:59:14.361 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T01:59:14.361 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:59:14.361 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:59:14.557 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T01:59:14.557 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T01:59:14.557 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T01:59:14.558 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:706: TEST_auto_repair_bluestore_failed_norecov: flush_pg_stats 2026-02-13T01:59:14.558 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T01:59:14.558 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T01:59:14.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T01:59:14.719 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:59:14.720 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T01:59:14.720 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T01:59:14.720 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:59:14.720 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T01:59:14.795 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855369 2026-02-13T01:59:14.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855369 2026-02-13T01:59:14.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-150323855369' 2026-02-13T01:59:14.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:59:14.796 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T01:59:14.871 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691846 2026-02-13T01:59:14.871 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691846 2026-02-13T01:59:14.871 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-150323855369 1-171798691846' 2026-02-13T01:59:14.871 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:59:14.871 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T01:59:14.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542159 2026-02-13T01:59:14.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542159 2026-02-13T01:59:14.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-150323855369 1-171798691846 2-60129542159' 2026-02-13T01:59:14.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:59:14.947 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-150323855369 2026-02-13T01:59:14.947 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:59:14.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T01:59:14.949 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-150323855369 2026-02-13T01:59:14.949 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:59:14.950 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855369 2026-02-13T01:59:14.951 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 150323855369 2026-02-13T01:59:14.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 150323855369' 2026-02-13T01:59:14.951 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:59:15.122 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855367 -lt 150323855369 2026-02-13T01:59:15.122 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:59:16.123 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T01:59:16.124 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:59:16.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855369 -lt 150323855369 2026-02-13T01:59:16.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:59:16.285 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-171798691846 2026-02-13T01:59:16.285 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:59:16.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T01:59:16.287 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-171798691846 2026-02-13T01:59:16.287 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:59:16.289 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691846 2026-02-13T01:59:16.289 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 171798691846' 2026-02-13T01:59:16.289 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 171798691846 2026-02-13T01:59:16.290 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T01:59:16.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691846 -lt 171798691846 2026-02-13T01:59:16.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:59:16.454 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542159 2026-02-13T01:59:16.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:59:16.455 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T01:59:16.456 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542159 2026-02-13T01:59:16.456 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:59:16.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542159 2026-02-13T01:59:16.458 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542159 2026-02-13T01:59:16.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542159' 2026-02-13T01:59:16.459 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T01:59:16.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542159 -lt 60129542159 2026-02-13T01:59:16.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_auto_repair_bluestore_failed_norecov: grep -q 'scrub_finish.*present with no repair possible' td/osd-scrub-repair/osd.1.log 2026-02-13T01:59:16.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:708: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2026-02-13T01:59:16.781 INFO:tasks.workunit.client.0.trial112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-13T01:59:16.781 INFO:tasks.workunit.client.0.trial112.stdout:1.0 10 0 0 0 0 56 0 0 10 0 10 active+clean+inconsistent+failed_repair 2026-02-13T01:59:10.722886+0000 19'10 41:105 [1,0] 1 [1,0] 1 19'10 2026-02-13T01:59:10.722825+0000 19'10 2026-02-13T01:59:10.722825+0000 0 0 periodic scrub scheduled @ 2026-02-14T01:59:10.722825+0000 10 0 2026-02-13T01:59:16.782 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:59:16.782 INFO:tasks.workunit.client.0.trial112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-13T01:59:16.782 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T01:59:16.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:709: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2026-02-13T01:59:16.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:709: TEST_auto_repair_bluestore_failed_norecov: grep -q '^1.0.*+failed_repair' 2026-02-13T01:59:16.956 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T01:59:16.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-13T01:59:16.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T01:59:16.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T01:59:16.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T01:59:16.966 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T01:59:16.966 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T01:59:16.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T01:59:16.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T01:59:16.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T01:59:17.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T01:59:17.077 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T01:59:17.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T01:59:17.079 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T01:59:17.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T01:59:17.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T01:59:17.081 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T01:59:17.082 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T01:59:17.082 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T01:59:17.082 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T01:59:17.082 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T01:59:17.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T01:59:17.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T01:59:17.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T01:59:17.097 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T01:59:17.097 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:59:17.097 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:59:17.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T01:59:17.099 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T01:59:17.099 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T01:59:17.099 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-13T01:59:17.099 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-13T01:59:17.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-13T01:59:17.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-13T01:59:17.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T01:59:17.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T01:59:17.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T01:59:17.100 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T01:59:17.100 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T01:59:17.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T01:59:17.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T01:59:17.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T01:59:17.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T01:59:17.101 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T01:59:17.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T01:59:17.103 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T01:59:17.104 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T01:59:17.104 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T01:59:17.105 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T01:59:17.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T01:59:17.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T01:59:17.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T01:59:17.107 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T01:59:17.108 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T01:59:17.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T01:59:17.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T01:59:17.110 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T01:59:17.110 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:59:17.110 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:59:17.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T01:59:17.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T01:59:17.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T01:59:17.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-13T01:59:17.113 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T01:59:17.113 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:59:17.113 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:59:17.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.35207 2026-02-13T01:59:17.115 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T01:59:17.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-13T01:59:17.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T01:59:17.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-13T01:59:17.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_scrub td/osd-scrub-repair 2026-02-13T01:59:17.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:542: TEST_auto_repair_bluestore_scrub: local dir=td/osd-scrub-repair 2026-02-13T01:59:17.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:543: TEST_auto_repair_bluestore_scrub: local poolname=testpool 2026-02-13T01:59:17.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:546: TEST_auto_repair_bluestore_scrub: run_mon td/osd-scrub-repair a 2026-02-13T01:59:17.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-13T01:59:17.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-13T01:59:17.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-13T01:59:17.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-13T01:59:17.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-13T01:59:17.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-13T01:59:17.138 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-13T01:59:17.138 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:59:17.138 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:59:17.138 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:59:17.138 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:59:17.138 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:59:17.139 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:59:17.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-13T01:59:17.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-13T01:59:17.169 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-13T01:59:17.169 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T01:59:17.169 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T01:59:17.170 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-13T01:59:17.171 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-13T01:59:17.171 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T01:59:17.171 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T01:59:17.171 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T01:59:17.171 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T01:59:17.171 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:59:17.171 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:59:17.171 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T01:59:17.172 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T01:59:17.172 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get fsid 2026-02-13T01:59:17.228 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-13T01:59:17.228 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T01:59:17.228 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T01:59:17.228 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-13T01:59:17.228 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-13T01:59:17.230 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T01:59:17.230 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T01:59:17.230 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T01:59:17.230 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T01:59:17.230 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:59:17.230 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:59:17.230 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T01:59:17.230 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T01:59:17.230 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get mon_host 2026-02-13T01:59:17.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:547: TEST_auto_repair_bluestore_scrub: run_mgr td/osd-scrub-repair x 2026-02-13T01:59:17.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-13T01:59:17.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-13T01:59:17.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-13T01:59:17.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-13T01:59:17.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-13T01:59:17.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-13T01:59:17.383 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-13T01:59:17.383 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:59:17.383 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:59:17.384 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:59:17.384 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:59:17.384 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:59:17.385 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:59:17.385 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T01:59:17.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T01:59:17.402 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:551: TEST_auto_repair_bluestore_scrub: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2026-02-13T01:59:17.403 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:552: TEST_auto_repair_bluestore_scrub: seq 0 2 2026-02-13T01:59:17.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:552: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2026-02-13T01:59:17.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:553: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2026-02-13T01:59:17.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T01:59:17.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T01:59:17.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-13T01:59:17.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T01:59:17.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T01:59:17.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T01:59:17.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T01:59:17.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T01:59:17.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T01:59:17.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T01:59:17.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T01:59:17.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T01:59:17.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T01:59:17.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T01:59:17.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T01:59:17.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:59:17.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:59:17.407 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:59:17.407 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:59:17.407 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:59:17.408 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:59:17.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:59:17.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T01:59:17.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T01:59:17.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T01:59:17.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T01:59:17.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T01:59:17.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T01:59:17.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T01:59:17.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T01:59:17.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T01:59:17.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T01:59:17.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2026-02-13T01:59:17.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T01:59:17.409 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T01:59:17.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7fc78b9f-7934-4333-8ea7-b1a60195dd17 2026-02-13T01:59:17.409 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 7fc78b9f-7934-4333-8ea7-b1a60195dd17 2026-02-13T01:59:17.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 7fc78b9f-7934-4333-8ea7-b1a60195dd17' 2026-02-13T01:59:17.409 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T01:59:17.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB1hY5p9DnwGBAAbrOJWVZ2e+RbOl8OLNeuCw== 2026-02-13T01:59:17.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB1hY5p9DnwGBAAbrOJWVZ2e+RbOl8OLNeuCw=="}' 2026-02-13T01:59:17.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7fc78b9f-7934-4333-8ea7-b1a60195dd17 -i td/osd-scrub-repair/0/new.json 2026-02-13T01:59:17.512 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T01:59:17.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-13T01:59:17.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQB1hY5p9DnwGBAAbrOJWVZ2e+RbOl8OLNeuCw== --osd-uuid 7fc78b9f-7934-4333-8ea7-b1a60195dd17 2026-02-13T01:59:17.538 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:59:17.530+0000 7fccf894da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:59:17.540 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:59:17.534+0000 7fccf894da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:59:17.541 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:59:17.534+0000 7fccf894da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:59:17.542 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:59:17.534+0000 7fccf894da40 -1 bdev(0x5566bcad0c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-13T01:59:17.542 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:59:17.534+0000 7fccf894da40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-13T01:59:18.254 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-13T01:59:18.254 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T01:59:18.255 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-13T01:59:18.256 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T01:59:18.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T01:59:18.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-13T01:59:18.362 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T01:59:18.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2026-02-13T01:59:18.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T01:59:18.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T01:59:18.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T01:59:18.390 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:59:18.382+0000 7ff2c7e28a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:59:18.392 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:59:18.386+0000 7ff2c7e28a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:59:18.393 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:59:18.386+0000 7ff2c7e28a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:59:18.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-13T01:59:18.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T01:59:18.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T01:59:18.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T01:59:18.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T01:59:18.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:59:18.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T01:59:18.458 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T01:59:18.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:59:18.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T01:59:18.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:59:18.766 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:59:18.758+0000 7ff2c7e28a40 -1 Falling back to public interface 2026-02-13T01:59:19.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:59:19.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:59:19.551 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T01:59:19.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T01:59:19.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:59:19.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T01:59:19.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:59:19.780 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:59:19.770+0000 7ff2c7e28a40 -1 osd.0 0 log_to_monitors true 2026-02-13T01:59:20.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:59:20.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:59:20.642 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T01:59:20.642 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T01:59:20.642 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:59:20.642 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T01:59:20.803 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3123198817,v1:127.0.0.1:6801/3123198817] [v2:127.0.0.1:6802/3123198817,v1:127.0.0.1:6803/3123198817] exists,up 7fc78b9f-7934-4333-8ea7-b1a60195dd17 2026-02-13T01:59:20.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T01:59:20.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T01:59:20.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T01:59:20.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:552: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2026-02-13T01:59:20.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:553: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2026-02-13T01:59:20.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T01:59:20.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T01:59:20.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-13T01:59:20.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T01:59:20.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T01:59:20.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T01:59:20.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T01:59:20.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T01:59:20.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T01:59:20.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T01:59:20.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T01:59:20.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T01:59:20.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T01:59:20.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T01:59:20.808 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T01:59:20.808 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:59:20.809 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:59:20.809 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:59:20.809 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:59:20.809 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:59:20.809 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:59:20.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:59:20.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T01:59:20.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T01:59:20.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T01:59:20.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T01:59:20.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T01:59:20.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T01:59:20.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T01:59:20.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T01:59:20.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T01:59:20.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T01:59:20.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2026-02-13T01:59:20.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T01:59:20.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T01:59:20.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ceeeabde-22a5-40ce-bda8-1de382f622e6 2026-02-13T01:59:20.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 ceeeabde-22a5-40ce-bda8-1de382f622e6' 2026-02-13T01:59:20.811 INFO:tasks.workunit.client.0.trial112.stdout:add osd1 ceeeabde-22a5-40ce-bda8-1de382f622e6 2026-02-13T01:59:20.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T01:59:20.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB4hY5pBJTeMBAA6dY3Yqlpuu+V+OShAwG8jQ== 2026-02-13T01:59:20.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB4hY5pBJTeMBAA6dY3Yqlpuu+V+OShAwG8jQ=="}' 2026-02-13T01:59:20.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ceeeabde-22a5-40ce-bda8-1de382f622e6 -i td/osd-scrub-repair/1/new.json 2026-02-13T01:59:20.987 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T01:59:20.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-13T01:59:20.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQB4hY5pBJTeMBAA6dY3Yqlpuu+V+OShAwG8jQ== --osd-uuid ceeeabde-22a5-40ce-bda8-1de382f622e6 2026-02-13T01:59:21.017 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:59:21.014+0000 7fa75269da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:59:21.018 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:59:21.014+0000 7fa75269da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:59:21.019 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:59:21.014+0000 7fa75269da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:59:21.020 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:59:21.014+0000 7fa75269da40 -1 bdev(0x558302494c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-13T01:59:21.020 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:59:21.014+0000 7fa75269da40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-13T01:59:21.723 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-13T01:59:21.723 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T01:59:21.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-13T01:59:21.724 INFO:tasks.workunit.client.0.trial112.stdout:adding osd1 key to auth repository 2026-02-13T01:59:21.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T01:59:21.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-13T01:59:21.932 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T01:59:21.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2026-02-13T01:59:21.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T01:59:21.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T01:59:21.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T01:59:21.958 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:59:21.954+0000 7f61e859aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:59:21.960 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:59:21.958+0000 7f61e859aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:59:21.961 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:59:21.958+0000 7f61e859aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:59:22.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-13T01:59:22.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T01:59:22.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T01:59:22.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T01:59:22.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T01:59:22.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:59:22.096 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T01:59:22.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T01:59:22.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:59:22.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T01:59:22.255 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:59:22.330 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:59:22.326+0000 7f61e859aa40 -1 Falling back to public interface 2026-02-13T01:59:22.910 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:59:22.906+0000 7f61e859aa40 -1 osd.1 0 log_to_monitors true 2026-02-13T01:59:23.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:59:23.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:59:23.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T01:59:23.257 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T01:59:23.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:59:23.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T01:59:23.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:59:24.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:59:24.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:59:24.429 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T01:59:24.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T01:59:24.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:59:24.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T01:59:24.587 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4165018950,v1:127.0.0.1:6811/4165018950] [v2:127.0.0.1:6812/4165018950,v1:127.0.0.1:6813/4165018950] exists,up ceeeabde-22a5-40ce-bda8-1de382f622e6 2026-02-13T01:59:24.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T01:59:24.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T01:59:24.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T01:59:24.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:552: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2026-02-13T01:59:24.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:553: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2026-02-13T01:59:24.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T01:59:24.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T01:59:24.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-13T01:59:24.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T01:59:24.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-13T01:59:24.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T01:59:24.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T01:59:24.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T01:59:24.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T01:59:24.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-13T01:59:24.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-13T01:59:24.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T01:59:24.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T01:59:24.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T01:59:24.592 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T01:59:24.592 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:59:24.592 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:59:24.592 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:59:24.592 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:59:24.592 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:59:24.592 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:59:24.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:59:24.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T01:59:24.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T01:59:24.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T01:59:24.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T01:59:24.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T01:59:24.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T01:59:24.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T01:59:24.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T01:59:24.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T01:59:24.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T01:59:24.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2026-02-13T01:59:24.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-13T01:59:24.594 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T01:59:24.594 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a61cbc02-7c95-4393-9afc-00f5b55248e8 2026-02-13T01:59:24.594 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 a61cbc02-7c95-4393-9afc-00f5b55248e8' 2026-02-13T01:59:24.595 INFO:tasks.workunit.client.0.trial112.stdout:add osd2 a61cbc02-7c95-4393-9afc-00f5b55248e8 2026-02-13T01:59:24.595 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T01:59:24.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB8hY5pjobTIxAAuFrV2Oim/JYFUkQT4wlqFQ== 2026-02-13T01:59:24.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB8hY5pjobTIxAAuFrV2Oim/JYFUkQT4wlqFQ=="}' 2026-02-13T01:59:24.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a61cbc02-7c95-4393-9afc-00f5b55248e8 -i td/osd-scrub-repair/2/new.json 2026-02-13T01:59:24.771 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T01:59:24.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-13T01:59:24.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQB8hY5pjobTIxAAuFrV2Oim/JYFUkQT4wlqFQ== --osd-uuid a61cbc02-7c95-4393-9afc-00f5b55248e8 2026-02-13T01:59:24.803 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:59:24.798+0000 7f9f4f060a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:59:24.804 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:59:24.802+0000 7f9f4f060a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:59:24.805 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:59:24.802+0000 7f9f4f060a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:59:24.806 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:59:24.802+0000 7f9f4f060a40 -1 bdev(0x55f352296c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-13T01:59:24.806 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:59:24.802+0000 7f9f4f060a40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-13T01:59:25.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-13T01:59:25.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T01:59:25.451 INFO:tasks.workunit.client.0.trial112.stdout:adding osd2 key to auth repository 2026-02-13T01:59:25.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-13T01:59:25.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T01:59:25.665 INFO:tasks.workunit.client.0.trial112.stdout:start osd.2 2026-02-13T01:59:25.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-13T01:59:25.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2026-02-13T01:59:25.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T01:59:25.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T01:59:25.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T01:59:25.690 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:59:25.686+0000 7f0f0a3d3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:59:25.692 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:59:25.686+0000 7f0f0a3d3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:59:25.693 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:59:25.690+0000 7f0f0a3d3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:59:25.823 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-13T01:59:25.823 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T01:59:25.823 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T01:59:25.823 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T01:59:25.823 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T01:59:25.823 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:59:25.823 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T01:59:25.824 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T01:59:25.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:59:25.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T01:59:25.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:59:26.101 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:59:26.098+0000 7f0f0a3d3a40 -1 Falling back to public interface 2026-02-13T01:59:26.660 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:59:26.654+0000 7f0f0a3d3a40 -1 osd.2 0 log_to_monitors true 2026-02-13T01:59:26.988 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:59:26.988 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:59:26.988 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T01:59:26.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T01:59:26.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:59:26.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T01:59:27.144 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:59:28.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:59:28.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:59:28.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T01:59:28.147 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T01:59:28.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:59:28.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T01:59:28.310 INFO:tasks.workunit.client.0.trial112.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4003580159,v1:127.0.0.1:6819/4003580159] [v2:127.0.0.1:6820/4003580159,v1:127.0.0.1:6821/4003580159] exists,up a61cbc02-7c95-4393-9afc-00f5b55248e8 2026-02-13T01:59:28.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T01:59:28.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T01:59:28.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T01:59:28.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:556: TEST_auto_repair_bluestore_scrub: create_pool testpool 1 1 2026-02-13T01:59:28.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create testpool 1 1 2026-02-13T01:59:28.555 INFO:tasks.workunit.client.0.trial112.stderr:pool 'testpool' created 2026-02-13T01:59:28.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T01:59:29.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:557: TEST_auto_repair_bluestore_scrub: ceph osd pool set testpool size 2 2026-02-13T01:59:29.745 INFO:tasks.workunit.client.0.trial112.stderr:set pool 1 size to 2 2026-02-13T01:59:29.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:558: TEST_auto_repair_bluestore_scrub: wait_for_clean 2026-02-13T01:59:29.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T01:59:29.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T01:59:29.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T01:59:29.758 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T01:59:29.759 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T01:59:29.759 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T01:59:29.759 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T01:59:29.759 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T01:59:29.759 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T01:59:29.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T01:59:29.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T01:59:29.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T01:59:29.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T01:59:29.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T01:59:29.875 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T01:59:30.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T01:59:30.035 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:59:30.035 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T01:59:30.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T01:59:30.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:59:30.035 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T01:59:30.108 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-13T01:59:30.108 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-13T01:59:30.108 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-13T01:59:30.108 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:59:30.109 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T01:59:30.180 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705667 2026-02-13T01:59:30.181 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705667 2026-02-13T01:59:30.181 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705667' 2026-02-13T01:59:30.181 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:59:30.181 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T01:59:30.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574850 2026-02-13T01:59:30.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574850 2026-02-13T01:59:30.258 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705667 2-55834574850' 2026-02-13T01:59:30.258 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:59:30.258 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-13T01:59:30.258 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:59:30.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T01:59:30.260 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-13T01:59:30.260 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:59:30.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-13T01:59:30.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-13T01:59:30.262 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836483 2026-02-13T01:59:30.262 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:59:30.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-13T01:59:30.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:59:31.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T01:59:31.431 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:59:31.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-13T01:59:31.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:59:32.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T01:59:32.593 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:59:32.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836483 2026-02-13T01:59:32.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:59:32.753 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705667 2026-02-13T01:59:32.753 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:59:32.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T01:59:32.755 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705667 2026-02-13T01:59:32.755 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:59:32.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705667 2026-02-13T01:59:32.757 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705667 2026-02-13T01:59:32.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705667' 2026-02-13T01:59:32.757 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T01:59:32.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705667 -lt 38654705667 2026-02-13T01:59:32.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:59:32.922 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574850 2026-02-13T01:59:32.922 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:59:32.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T01:59:32.924 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574850 2026-02-13T01:59:32.924 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:59:32.925 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574850 2026-02-13T01:59:32.925 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574850' 2026-02-13T01:59:32.926 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574850 2026-02-13T01:59:32.926 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T01:59:33.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574850 -lt 55834574850 2026-02-13T01:59:33.095 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T01:59:33.096 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:59:33.096 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:59:33.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T01:59:33.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T01:59:33.287 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T01:59:33.287 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T01:59:33.287 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T01:59:33.287 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T01:59:33.288 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T01:59:33.288 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T01:59:33.441 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T01:59:33.441 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T01:59:33.442 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:59:33.442 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:59:33.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T01:59:33.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T01:59:33.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T01:59:33.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_scrub: local payload=ABCDEF 2026-02-13T01:59:33.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:562: TEST_auto_repair_bluestore_scrub: echo ABCDEF 2026-02-13T01:59:33.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:563: TEST_auto_repair_bluestore_scrub: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-13T01:59:33.659 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:567: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2026-02-13T01:59:33.659 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-13T01:59:33.659 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-13T01:59:33.660 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2026-02-13T01:59:33.660 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-13T01:59:33.660 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-13T01:59:33.660 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-13T01:59:33.660 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T01:59:33.820 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-13T01:59:33.821 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2026-02-13T01:59:33.821 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-13T01:59:33.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:567: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2026-02-13T01:59:33.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T01:59:33.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T01:59:33.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T01:59:33.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T01:59:33.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2026-02-13T01:59:33.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T01:59:33.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T01:59:33.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T01:59:33.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T01:59:33.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T01:59:33.984 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T01:59:33.984 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T01:59:33.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T01:59:33.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T01:59:33.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T01:59:34.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T01:59:34.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2026-02-13T01:59:34.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T01:59:34.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T01:59:34.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T01:59:34.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T01:59:34.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T01:59:34.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2026-02-13T01:59:34.476 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:eb822e21:::SOMETHING:head# 2026-02-13T01:59:34.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2026-02-13T01:59:34.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T01:59:34.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T01:59:34.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T01:59:34.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T01:59:34.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T01:59:34.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T01:59:34.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T01:59:34.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T01:59:34.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T01:59:34.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T01:59:34.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T01:59:34.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T01:59:34.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T01:59:34.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T01:59:34.579 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T01:59:34.579 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:59:34.579 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:59:34.579 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:59:34.579 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:59:34.580 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:59:34.580 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:59:34.581 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:59:34.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T01:59:34.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T01:59:34.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T01:59:34.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T01:59:34.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T01:59:34.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T01:59:34.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T01:59:34.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T01:59:34.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2026-02-13T01:59:34.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T01:59:34.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T01:59:34.583 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T01:59:34.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2026-02-13T01:59:34.583 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T01:59:34.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T01:59:34.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T01:59:34.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T01:59:34.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T01:59:34.600 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:59:34.594+0000 7f42e8c23a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:59:34.601 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:59:34.598+0000 7f42e8c23a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:59:34.602 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:59:34.598+0000 7f42e8c23a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:59:34.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T01:59:34.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T01:59:34.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T01:59:34.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T01:59:34.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T01:59:34.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:59:34.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T01:59:34.741 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T01:59:34.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:59:34.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T01:59:34.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T01:59:34.954 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:59:34.950+0000 7f42e8c23a40 -1 Falling back to public interface 2026-02-13T01:59:35.551 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:59:35.546+0000 7f42e8c23a40 -1 osd.0 18 log_to_monitors true 2026-02-13T01:59:35.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T01:59:35.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:59:35.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T01:59:35.903 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:59:35.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:59:35.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T01:59:36.063 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 22 up_thru 0 down_at 19 last_clean_interval [5,18) [v2:127.0.0.1:6800/344095943,v1:127.0.0.1:6801/344095943] [v2:127.0.0.1:6802/344095943,v1:127.0.0.1:6803/344095943] exists,up 7fc78b9f-7934-4333-8ea7-b1a60195dd17 2026-02-13T01:59:36.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T01:59:36.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T01:59:36.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T01:59:36.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T01:59:36.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T01:59:36.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T01:59:36.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T01:59:36.064 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T01:59:36.064 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T01:59:36.064 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T01:59:36.065 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T01:59:36.065 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T01:59:36.065 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T01:59:36.167 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T01:59:36.167 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T01:59:36.167 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T01:59:36.167 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T01:59:36.167 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T01:59:36.168 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T01:59:36.339 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T01:59:36.340 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:59:36.340 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T01:59:36.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T01:59:36.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:59:36.340 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T01:59:36.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280514 2026-02-13T01:59:36.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280514 2026-02-13T01:59:36.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280514' 2026-02-13T01:59:36.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:59:36.412 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T01:59:36.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705669 2026-02-13T01:59:36.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705669 2026-02-13T01:59:36.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280514 1-38654705669' 2026-02-13T01:59:36.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:59:36.485 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T01:59:36.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574852 2026-02-13T01:59:36.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574852 2026-02-13T01:59:36.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280514 1-38654705669 2-55834574852' 2026-02-13T01:59:36.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:59:36.560 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-94489280514 2026-02-13T01:59:36.560 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:59:36.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T01:59:36.562 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-94489280514 2026-02-13T01:59:36.562 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:59:36.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280514 2026-02-13T01:59:36.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 94489280514' 2026-02-13T01:59:36.563 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 94489280514 2026-02-13T01:59:36.564 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:59:36.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 94489280514 2026-02-13T01:59:36.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:59:37.735 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T01:59:37.735 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:59:37.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280514 -lt 94489280514 2026-02-13T01:59:37.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:59:37.896 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705669 2026-02-13T01:59:37.896 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:59:37.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T01:59:37.898 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705669 2026-02-13T01:59:37.898 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:59:37.899 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705669 2026-02-13T01:59:37.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705669' 2026-02-13T01:59:37.900 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 38654705669 2026-02-13T01:59:37.900 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T01:59:38.066 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705669 -lt 38654705669 2026-02-13T01:59:38.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:59:38.067 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574852 2026-02-13T01:59:38.067 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:59:38.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T01:59:38.069 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574852 2026-02-13T01:59:38.070 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:59:38.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574852 2026-02-13T01:59:38.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574852' 2026-02-13T01:59:38.071 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 55834574852 2026-02-13T01:59:38.071 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T01:59:38.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574852 -lt 55834574852 2026-02-13T01:59:38.240 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T01:59:38.240 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:59:38.241 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:59:38.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T01:59:38.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T01:59:38.431 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T01:59:38.431 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T01:59:38.431 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T01:59:38.431 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T01:59:38.431 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T01:59:38.431 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T01:59:38.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-13T01:59:38.590 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T01:59:38.590 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:59:38.590 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:59:38.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-13T01:59:38.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' -1 2026-02-13T01:59:38.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-13T01:59:38.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=0 2026-02-13T01:59:38.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-13T01:59:38.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-13T01:59:38.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-13T01:59:38.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T01:59:38.890 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T01:59:38.890 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T01:59:38.890 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T01:59:38.890 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T01:59:38.891 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T01:59:38.891 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T01:59:39.047 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-13T01:59:39.048 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T01:59:39.048 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:59:39.048 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:59:39.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-13T01:59:39.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-13T01:59:39.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-13T01:59:39.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-13T01:59:39.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-13T01:59:39.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-13T01:59:39.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-13T01:59:39.246 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-13T01:59:39.246 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-13T01:59:39.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-13T01:59:39.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-13T01:59:39.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-13T01:59:39.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-13T01:59:39.439 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-13T01:59:39.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-13T01:59:39.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T01:59:39.640 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T01:59:39.640 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T01:59:39.641 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T01:59:39.641 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T01:59:39.641 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T01:59:39.641 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T01:59:39.799 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T01:59:39.799 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T01:59:39.799 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:59:39.800 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:59:39.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T01:59:39.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T01:59:39.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T01:59:39.991 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_scrub: get_pg testpool SOMETHING 2026-02-13T01:59:39.991 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2026-02-13T01:59:39.991 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-13T01:59:39.991 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool SOMETHING 2026-02-13T01:59:39.991 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T01:59:40.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_scrub: local pgid=1.0 2026-02-13T01:59:40.149 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:570: TEST_auto_repair_bluestore_scrub: get_primary testpool SOMETHING 2026-02-13T01:59:40.149 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-13T01:59:40.149 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-13T01:59:40.149 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-13T01:59:40.149 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T01:59:40.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:570: TEST_auto_repair_bluestore_scrub: local primary=1 2026-02-13T01:59:40.314 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_scrub: get_last_scrub_stamp 1.0 2026-02-13T01:59:40.314 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T01:59:40.314 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T01:59:40.314 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T01:59:40.315 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T01:59:40.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_scrub: local last_scrub_stamp=2026-02-13T01:59:28.550844+0000 2026-02-13T01:59:40.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:572: TEST_auto_repair_bluestore_scrub: ceph tell 1.0 schedule-scrub 2026-02-13T01:59:40.541 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T01:59:40.541 INFO:tasks.workunit.client.0.trial112.stdout: "deep": false, 2026-02-13T01:59:40.541 INFO:tasks.workunit.client.0.trial112.stdout: "must": false, 2026-02-13T01:59:40.541 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "2026-02-11T01:59:40.539631+0000" 2026-02-13T01:59:40.542 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T01:59:40.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:575: TEST_auto_repair_bluestore_scrub: wait_for_scrub 1.0 2026-02-13T01:59:28.550844+0000 2026-02-13T01:59:40.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-13T01:59:40.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T01:59:28.550844+0000 2026-02-13T01:59:40.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T01:59:40.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T01:59:40.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T01:59:40.550 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T01:59:40.550 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T01:59:40.550 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T01:59:40.550 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T01:59:40.550 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T01:59:40.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T01:59:28.550844+0000 '>' 2026-02-13T01:59:28.550844+0000 2026-02-13T01:59:40.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T01:59:41.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T01:59:41.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T01:59:41.716 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T01:59:41.717 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T01:59:41.717 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T01:59:41.717 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T01:59:41.717 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T01:59:41.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T01:59:28.550844+0000 '>' 2026-02-13T01:59:28.550844+0000 2026-02-13T01:59:41.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T01:59:42.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T01:59:42.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T01:59:42.876 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T01:59:42.876 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T01:59:42.876 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T01:59:42.877 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T01:59:42.877 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T01:59:43.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T01:59:28.550844+0000 '>' 2026-02-13T01:59:28.550844+0000 2026-02-13T01:59:43.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T01:59:44.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T01:59:44.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T01:59:44.033 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T01:59:44.033 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T01:59:44.033 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T01:59:44.034 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T01:59:44.034 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T01:59:44.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T01:59:28.550844+0000 '>' 2026-02-13T01:59:28.550844+0000 2026-02-13T01:59:44.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T01:59:45.190 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T01:59:45.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T01:59:45.191 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T01:59:45.191 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T01:59:45.191 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T01:59:45.192 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T01:59:45.192 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T01:59:45.347 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T01:59:28.550844+0000 '>' 2026-02-13T01:59:28.550844+0000 2026-02-13T01:59:45.348 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T01:59:46.348 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T01:59:46.348 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T01:59:46.349 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T01:59:46.349 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T01:59:46.349 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T01:59:46.350 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T01:59:46.350 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T01:59:46.509 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T01:59:41.715666+0000 '>' 2026-02-13T01:59:28.550844+0000 2026-02-13T01:59:46.509 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T01:59:46.509 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:576: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2026-02-13T01:59:46.658 INFO:tasks.workunit.client.0.trial112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-13T01:59:46.658 INFO:tasks.workunit.client.0.trial112.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2026-02-13T01:59:41.717165+0000 18'1 24:69 [1,0] 1 [1,0] 1 18'1 2026-02-13T01:59:41.715666+0000 18'1 2026-02-13T01:59:41.715666+0000 0 1 periodic scrub scheduled @ 2026-02-14T01:59:41.715666+0000 1 0 2026-02-13T01:59:46.658 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:59:46.658 INFO:tasks.workunit.client.0.trial112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-13T01:59:46.658 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T01:59:46.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:578: TEST_auto_repair_bluestore_scrub: sleep 2 2026-02-13T01:59:48.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:579: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2026-02-13T01:59:48.815 INFO:tasks.workunit.client.0.trial112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-13T01:59:48.815 INFO:tasks.workunit.client.0.trial112.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2026-02-13T01:59:41.717165+0000 18'1 24:69 [1,0] 1 [1,0] 1 18'1 2026-02-13T01:59:41.715666+0000 18'1 2026-02-13T01:59:41.715666+0000 0 1 periodic scrub scheduled @ 2026-02-14T01:59:41.715666+0000 1 0 2026-02-13T01:59:48.815 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:59:48.815 INFO:tasks.workunit.client.0.trial112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-13T01:59:48.816 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T01:59:48.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:580: TEST_auto_repair_bluestore_scrub: sleep 2 2026-02-13T01:59:50.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:581: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2026-02-13T01:59:50.972 INFO:tasks.workunit.client.0.trial112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-13T01:59:50.973 INFO:tasks.workunit.client.0.trial112.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2026-02-13T01:59:41.717165+0000 18'1 24:69 [1,0] 1 [1,0] 1 18'1 2026-02-13T01:59:41.715666+0000 18'1 2026-02-13T01:59:41.715666+0000 0 1 periodic scrub scheduled @ 2026-02-14T01:59:41.715666+0000 1 0 2026-02-13T01:59:50.973 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:59:50.973 INFO:tasks.workunit.client.0.trial112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-13T01:59:50.973 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T01:59:50.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:582: TEST_auto_repair_bluestore_scrub: sleep 5 2026-02-13T01:59:55.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:583: TEST_auto_repair_bluestore_scrub: wait_for_clean 2026-02-13T01:59:55.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T01:59:55.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T01:59:55.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T01:59:55.984 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T01:59:55.985 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T01:59:55.985 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T01:59:55.985 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T01:59:55.985 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T01:59:55.986 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T01:59:56.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T01:59:56.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T01:59:56.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T01:59:56.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T01:59:56.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T01:59:56.084 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T01:59:56.253 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T01:59:56.253 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T01:59:56.253 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T01:59:56.253 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T01:59:56.253 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:59:56.253 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T01:59:56.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280519 2026-02-13T01:59:56.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280519 2026-02-13T01:59:56.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280519' 2026-02-13T01:59:56.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:59:56.327 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T01:59:56.401 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705674 2026-02-13T01:59:56.401 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705674 2026-02-13T01:59:56.401 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280519 1-38654705674' 2026-02-13T01:59:56.401 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T01:59:56.401 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T01:59:56.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574857 2026-02-13T01:59:56.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574857 2026-02-13T01:59:56.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280519 1-38654705674 2-55834574857' 2026-02-13T01:59:56.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:59:56.480 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-94489280519 2026-02-13T01:59:56.480 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:59:56.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T01:59:56.482 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-94489280519 2026-02-13T01:59:56.482 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:59:56.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280519 2026-02-13T01:59:56.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 94489280519' 2026-02-13T01:59:56.484 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 94489280519 2026-02-13T01:59:56.485 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:59:56.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280517 -lt 94489280519 2026-02-13T01:59:56.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T01:59:57.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T01:59:57.649 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T01:59:57.806 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280519 -lt 94489280519 2026-02-13T01:59:57.806 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:59:57.807 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705674 2026-02-13T01:59:57.807 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:59:57.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T01:59:57.809 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705674 2026-02-13T01:59:57.809 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:59:57.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705674 2026-02-13T01:59:57.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705674' 2026-02-13T01:59:57.811 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705674 2026-02-13T01:59:57.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T01:59:57.981 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705674 -lt 38654705674 2026-02-13T01:59:57.981 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T01:59:57.981 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574857 2026-02-13T01:59:57.982 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T01:59:57.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T01:59:57.984 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574857 2026-02-13T01:59:57.984 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T01:59:57.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574857 2026-02-13T01:59:57.985 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574857 2026-02-13T01:59:57.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574857' 2026-02-13T01:59:57.986 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T01:59:58.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574857 -lt 55834574857 2026-02-13T01:59:58.147 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T01:59:58.148 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:59:58.148 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:59:58.343 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T01:59:58.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T01:59:58.345 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T01:59:58.345 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T01:59:58.345 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T01:59:58.345 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T01:59:58.345 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T01:59:58.345 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T01:59:58.506 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T01:59:58.507 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T01:59:58.507 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T01:59:58.507 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T01:59:58.705 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T01:59:58.705 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T01:59:58.705 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T01:59:58.705 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:584: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2026-02-13T01:59:58.858 INFO:tasks.workunit.client.0.trial112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-13T01:59:58.859 INFO:tasks.workunit.client.0.trial112.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2026-02-13T01:59:41.717165+0000 18'1 24:69 [1,0] 1 [1,0] 1 18'1 2026-02-13T01:59:41.715666+0000 18'1 2026-02-13T01:59:41.715666+0000 0 1 periodic scrub scheduled @ 2026-02-14T01:59:41.715666+0000 1 0 2026-02-13T01:59:58.859 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T01:59:58.859 INFO:tasks.workunit.client.0.trial112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-13T01:59:58.859 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T01:59:58.868 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:587: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2026-02-13T01:59:58.869 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-13T01:59:58.869 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-13T01:59:58.869 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2026-02-13T01:59:58.869 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-13T01:59:58.869 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-13T01:59:58.870 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-13T01:59:58.870 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T01:59:59.027 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-13T01:59:59.028 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2026-02-13T01:59:59.028 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-13T01:59:59.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:587: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-13T01:59:59.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T01:59:59.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T01:59:59.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T01:59:59.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T01:59:59.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-13T01:59:59.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T01:59:59.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T01:59:59.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T01:59:59.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T01:59:59.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T01:59:59.192 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T01:59:59.193 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T01:59:59.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T01:59:59.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T01:59:59.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T01:59:59.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T01:59:59.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-13T01:59:59.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T01:59:59.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T01:59:59.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T01:59:59.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T01:59:59.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T01:59:59.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2026-02-13T01:59:59.571 INFO:tasks.workunit.client.0.trial112.stdout:_ 2026-02-13T01:59:59.571 INFO:tasks.workunit.client.0.trial112.stdout:snapset 2026-02-13T01:59:59.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2026-02-13T01:59:59.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T01:59:59.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T01:59:59.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T01:59:59.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T01:59:59.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T01:59:59.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T01:59:59.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T01:59:59.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T01:59:59.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T01:59:59.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T01:59:59.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T01:59:59.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T01:59:59.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T01:59:59.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T01:59:59.656 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T01:59:59.656 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T01:59:59.656 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T01:59:59.656 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T01:59:59.657 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T01:59:59.657 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T01:59:59.657 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:59:59.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T01:59:59.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T01:59:59.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T01:59:59.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T01:59:59.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T01:59:59.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T01:59:59.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T01:59:59.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T01:59:59.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T01:59:59.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2026-02-13T01:59:59.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T01:59:59.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T01:59:59.659 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T01:59:59.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2026-02-13T01:59:59.659 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T01:59:59.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T01:59:59.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T01:59:59.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T01:59:59.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T01:59:59.676 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:59:59.670+0000 7feea8e8ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:59:59.677 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:59:59.674+0000 7feea8e8ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:59:59.678 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T01:59:59.674+0000 7feea8e8ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T01:59:59.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T01:59:59.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T01:59:59.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T01:59:59.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T01:59:59.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T01:59:59.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T01:59:59.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T01:59:59.821 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T01:59:59.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T01:59:59.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T01:59:59.981 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:00:00.126 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:00.122+0000 7feea8e8ba40 -1 Falling back to public interface 2026-02-13T02:00:00.953 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:00.950+0000 7feea8e8ba40 -1 osd.0 25 log_to_monitors true 2026-02-13T02:00:00.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:00:00.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:00:00.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:00:00.983 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:00:00.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:00:00.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:00:01.141 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 28 up_thru 0 down_at 25 last_clean_interval [22,25) [v2:127.0.0.1:6800/1498467342,v1:127.0.0.1:6801/1498467342] [v2:127.0.0.1:6802/1498467342,v1:127.0.0.1:6803/1498467342] exists,up 7fc78b9f-7934-4333-8ea7-b1a60195dd17 2026-02-13T02:00:01.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:00:01.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:00:01.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:00:01.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:00:01.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:00:01.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:00:01.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:00:01.142 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:00:01.142 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:00:01.143 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:00:01.143 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:00:01.143 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:00:01.143 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:00:01.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:00:01.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:00:01.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:00:01.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:00:01.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:00:01.252 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:00:01.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:00:01.417 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:00:01.417 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:00:01.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:00:01.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:00:01.418 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:00:01.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=120259084290 2026-02-13T02:00:01.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 120259084290 2026-02-13T02:00:01.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-120259084290' 2026-02-13T02:00:01.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:00:01.492 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:00:01.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705676 2026-02-13T02:00:01.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705676 2026-02-13T02:00:01.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-120259084290 1-38654705676' 2026-02-13T02:00:01.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:00:01.565 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:00:01.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574859 2026-02-13T02:00:01.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574859 2026-02-13T02:00:01.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-120259084290 1-38654705676 2-55834574859' 2026-02-13T02:00:01.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:00:01.637 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-120259084290 2026-02-13T02:00:01.637 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:00:01.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:00:01.639 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-120259084290 2026-02-13T02:00:01.639 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:00:01.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=120259084290 2026-02-13T02:00:01.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 120259084290' 2026-02-13T02:00:01.640 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 120259084290 2026-02-13T02:00:01.641 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:00:01.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 120259084290 -lt 120259084290 2026-02-13T02:00:01.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:00:01.796 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705676 2026-02-13T02:00:01.797 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:00:01.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:00:01.799 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705676 2026-02-13T02:00:01.799 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:00:01.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705676 2026-02-13T02:00:01.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705676' 2026-02-13T02:00:01.800 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 38654705676 2026-02-13T02:00:01.801 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:00:01.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705676 -lt 38654705676 2026-02-13T02:00:01.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:00:01.967 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574859 2026-02-13T02:00:01.967 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:00:01.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:00:01.969 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574859 2026-02-13T02:00:01.969 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:00:01.970 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574859 2026-02-13T02:00:01.970 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574859' 2026-02-13T02:00:01.970 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 55834574859 2026-02-13T02:00:01.971 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:00:02.132 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574858 -lt 55834574859 2026-02-13T02:00:02.132 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:00:03.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:00:03.133 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:00:03.295 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574858 -lt 55834574859 2026-02-13T02:00:03.295 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:00:04.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:00:04.297 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:00:04.455 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574860 -lt 55834574859 2026-02-13T02:00:04.456 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:00:04.456 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:00:04.456 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:00:04.645 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:00:04.645 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:00:04.646 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:00:04.646 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:00:04.646 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:00:04.646 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:00:04.646 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:00:04.646 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:00:04.806 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-13T02:00:04.807 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:00:04.807 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:00:04.807 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:00:05.001 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-13T02:00:05.001 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' -1 2026-02-13T02:00:05.001 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-13T02:00:05.001 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=0 2026-02-13T02:00:05.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-13T02:00:05.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-13T02:00:05.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-13T02:00:05.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:00:05.103 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:00:05.103 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:00:05.103 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:00:05.103 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:00:05.104 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:00:05.104 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:00:05.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-13T02:00:05.261 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:00:05.262 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:00:05.262 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:00:05.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-13T02:00:05.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-13T02:00:05.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-13T02:00:05.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-13T02:00:05.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-13T02:00:05.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-13T02:00:05.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-13T02:00:05.457 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-13T02:00:05.457 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-13T02:00:05.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-13T02:00:05.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-13T02:00:05.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-13T02:00:05.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-13T02:00:05.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-13T02:00:05.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-13T02:00:05.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:00:05.850 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:00:05.850 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:00:05.850 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:00:05.851 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:00:05.851 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:00:05.851 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:00:06.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:00:06.007 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:00:06.008 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:00:06.008 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:00:06.199 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:00:06.199 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:00:06.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:00:06.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:588: TEST_auto_repair_bluestore_scrub: rados --pool testpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-13T02:00:06.219 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:589: TEST_auto_repair_bluestore_scrub: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-13T02:00:06.219 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:590: TEST_auto_repair_bluestore_scrub: grep scrub_finish td/osd-scrub-repair/osd.1.log 2026-02-13T02:00:06.222 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:59:40.662+0000 7f61c9d1f640 10 osd.1 pg_epoch: 24 pg[1.0( v 18'1 (0'0,18'1] local-lis/les=22/24 n=1 ec=14/14 lis/c=22/22 les/c/f=24/24/0 sis=22) [1,0] r=0 lpr=22 crt=18'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing(periodic-regular,deep-scrub-on-error)] scrubber: scrub_finish before flags: DEEP_SCRUB_ON_ERROR. repair state: no-repair. deep_scrub_on_error: 1 2026-02-13T02:00:06.222 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:59:40.662+0000 7f61c9d1f640 10 osd.1 pg_epoch: 24 pg[1.0( v 18'1 (0'0,18'1] local-lis/les=22/24 n=1 ec=14/14 lis/c=22/22 les/c/f=24/24/0 sis=22) [1,0] r=0 lpr=22 crt=18'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing(periodic-regular,deep-scrub-on-error)] scrubber: scrub_finish: will initiate a deep scrub to fix 1 errors 2026-02-13T02:00:06.222 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:59:40.662+0000 7f61c9d1f640 10 osd.1 pg_epoch: 24 pg[1.0( v 18'1 (0'0,18'1] local-lis/les=22/24 n=1 ec=14/14 lis/c=22/22 les/c/f=24/24/0 sis=22) [1,0] r=0 lpr=22 crt=18'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing(periodic-regular)] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2026-02-13T02:00:06.222 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:59:41.714+0000 7f61c5d17640 10 osd.1 pg_epoch: 24 pg[1.0( v 18'1 (0'0,18'1] local-lis/les=22/24 n=1 ec=14/14 lis/c=22/22 les/c/f=24/24/0 sis=22) [1,0] r=0 lpr=22 crt=18'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+repair(repairing,auto-repair)] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: repair. deep_scrub_on_error: 0 2026-02-13T02:00:06.222 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:59:41.714+0000 7f61c5d17640 10 osd.1 pg_epoch: 24 pg[1.0( v 18'1 (0'0,18'1] local-lis/les=22/24 n=1 ec=14/14 lis/c=22/22 les/c/f=24/24/0 sis=22) [1,0] r=0 lpr=22 crt=18'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+repair(repairing,auto-repair)] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2026-02-13T02:00:06.222 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:59:41.714+0000 7f61c5d17640 15 osd.1 pg_epoch: 24 pg[1.0( v 18'1 (0'0,18'1] local-lis/les=22/24 n=1 ec=14/14 lis/c=22/22 les/c/f=24/24/0 sis=22) [1,0] r=0 lpr=22 crt=18'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair(repairing,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2026-02-13T02:00:06.222 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:59:41.714+0000 7f61c5d17640 20 osd.1 pg_epoch: 24 pg[1.0( v 18'1 (0'0,18'1] local-lis/les=22/24 n=1 ec=14/14 lis/c=22/22 les/c/f=24/24/0 sis=22) [1,0] r=0 lpr=22 crt=18'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair(repairing,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2026-02-13T02:00:06.222 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T01:59:41.714+0000 7f61c5d17640 19 osd.1 pg_epoch: 24 pg[1.0( v 18'1 (0'0,18'1] local-lis/les=22/24 n=1 ec=14/14 lis/c=22/22 les/c/f=24/24/0 sis=22) [1,0] r=0 lpr=22 crt=18'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair(repairing,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2026-02-13T02:00:06.223 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_scrub: ceph pg 1.0 query 2026-02-13T02:00:06.223 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_scrub: jq .info.stats.stat_sum.num_objects_repaired 2026-02-13T02:00:06.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_scrub: COUNT=1 2026-02-13T02:00:06.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:594: TEST_auto_repair_bluestore_scrub: test 1 = 1 2026-02-13T02:00:06.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-13T02:00:06.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:00:06.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:00:06.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:00:06.298 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:00:06.298 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:00:06.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:00:06.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:00:06.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:00:06.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:00:06.408 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:00:06.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:00:06.409 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:00:06.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:00:06.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:00:06.411 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:00:06.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:00:06.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:00:06.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:00:06.413 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:00:06.414 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:00:06.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:00:06.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:00:06.427 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:00:06.427 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:00:06.428 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:00:06.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:00:06.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:00:06.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:00:06.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-13T02:00:06.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-13T02:00:06.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-13T02:00:06.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-13T02:00:06.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:00:06.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:00:06.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:00:06.431 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:00:06.431 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:00:06.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:00:06.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:00:06.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:00:06.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:00:06.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:00:06.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:00:06.435 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:00:06.435 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:00:06.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:00:06.436 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:00:06.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:00:06.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:00:06.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:00:06.438 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:00:06.439 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:00:06.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:00:06.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:00:06.442 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:00:06.442 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:00:06.442 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:00:06.442 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:00:06.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:00:06.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:00:06.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-13T02:00:06.445 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T02:00:06.445 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:00:06.445 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:00:06.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.35207 2026-02-13T02:00:06.447 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T02:00:06.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-13T02:00:06.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T02:00:06.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-13T02:00:06.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_tag td/osd-scrub-repair 2026-02-13T02:00:06.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:443: TEST_auto_repair_bluestore_tag: local dir=td/osd-scrub-repair 2026-02-13T02:00:06.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:444: TEST_auto_repair_bluestore_tag: local poolname=testpool 2026-02-13T02:00:06.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:447: TEST_auto_repair_bluestore_tag: run_mon td/osd-scrub-repair a 2026-02-13T02:00:06.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-13T02:00:06.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-13T02:00:06.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-13T02:00:06.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-13T02:00:06.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-13T02:00:06.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-13T02:00:06.469 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-13T02:00:06.470 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:00:06.470 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:00:06.471 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:00:06.471 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:00:06.471 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:00:06.471 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:00:06.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-13T02:00:06.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-13T02:00:06.501 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-13T02:00:06.501 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:00:06.502 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:00:06.502 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-13T02:00:06.503 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-13T02:00:06.503 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:00:06.503 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:00:06.503 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:00:06.503 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:00:06.503 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:00:06.503 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:00:06.503 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:00:06.504 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:00:06.504 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get fsid 2026-02-13T02:00:06.561 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-13T02:00:06.561 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:00:06.561 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:00:06.561 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-13T02:00:06.562 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-13T02:00:06.563 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:00:06.563 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:00:06.563 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:00:06.563 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:00:06.563 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:00:06.563 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:00:06.563 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:00:06.564 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:00:06.564 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get mon_host 2026-02-13T02:00:06.619 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:448: TEST_auto_repair_bluestore_tag: run_mgr td/osd-scrub-repair x 2026-02-13T02:00:06.619 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-13T02:00:06.619 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-13T02:00:06.619 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-13T02:00:06.619 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-13T02:00:06.620 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-13T02:00:06.620 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-13T02:00:06.718 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-13T02:00:06.718 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:00:06.718 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:00:06.719 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:00:06.719 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:00:06.719 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:00:06.719 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:00:06.720 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:00:06.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:00:06.739 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:455: TEST_auto_repair_bluestore_tag: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2026-02-13T02:00:06.739 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:456: TEST_auto_repair_bluestore_tag: seq 0 2 2026-02-13T02:00:06.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:456: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2026-02-13T02:00:06.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:457: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2026-02-13T02:00:06.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:00:06.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:00:06.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-13T02:00:06.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:00:06.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:00:06.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:00:06.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:00:06.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:00:06.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:00:06.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:00:06.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:00:06.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:00:06.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:00:06.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:00:06.744 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:00:06.744 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:00:06.744 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:00:06.744 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:00:06.744 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:00:06.744 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:00:06.744 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:00:06.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:00:06.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:00:06.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:00:06.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:00:06.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:00:06.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:00:06.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:00:06.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:00:06.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:00:06.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:00:06.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:00:06.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2026-02-13T02:00:06.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:00:06.746 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:00:06.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=39d03e9f-959e-4ffa-b825-1766b89b0da8 2026-02-13T02:00:06.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 39d03e9f-959e-4ffa-b825-1766b89b0da8' 2026-02-13T02:00:06.746 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 39d03e9f-959e-4ffa-b825-1766b89b0da8 2026-02-13T02:00:06.747 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:00:06.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCmhY5pF5zfLBAAFR+XWl3zNgnYz0jHl78rXg== 2026-02-13T02:00:06.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCmhY5pF5zfLBAAFR+XWl3zNgnYz0jHl78rXg=="}' 2026-02-13T02:00:06.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 39d03e9f-959e-4ffa-b825-1766b89b0da8 -i td/osd-scrub-repair/0/new.json 2026-02-13T02:00:06.845 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:00:06.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-13T02:00:06.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQCmhY5pF5zfLBAAFR+XWl3zNgnYz0jHl78rXg== --osd-uuid 39d03e9f-959e-4ffa-b825-1766b89b0da8 2026-02-13T02:00:06.871 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:06.866+0000 7fa444f6aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:06.873 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:06.870+0000 7fa444f6aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:06.873 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:06.870+0000 7fa444f6aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:06.874 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:06.870+0000 7fa444f6aa40 -1 bdev(0x561cab690c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-13T02:00:06.874 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:06.870+0000 7fa444f6aa40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-13T02:00:07.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-13T02:00:07.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:00:07.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-13T02:00:07.565 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T02:00:07.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:00:07.659 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T02:00:07.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-13T02:00:07.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2026-02-13T02:00:07.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:00:07.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:00:07.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:00:07.677 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:07.674+0000 7fcba5282a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:07.678 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:07.674+0000 7fcba5282a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:07.679 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:07.674+0000 7fcba5282a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:07.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-13T02:00:07.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:00:07.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:00:07.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:00:07.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:00:07.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:00:07.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:00:07.759 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:00:07.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:00:07.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:00:07.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:00:08.062 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:08.058+0000 7fcba5282a40 -1 Falling back to public interface 2026-02-13T02:00:08.689 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:08.686+0000 7fcba5282a40 -1 osd.0 0 log_to_monitors true 2026-02-13T02:00:08.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:00:08.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:00:08.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:00:08.853 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:00:08.854 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:00:08.854 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:00:08.971 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1728035069,v1:127.0.0.1:6801/1728035069] [v2:127.0.0.1:6802/1728035069,v1:127.0.0.1:6803/1728035069] exists,up 39d03e9f-959e-4ffa-b825-1766b89b0da8 2026-02-13T02:00:08.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:00:08.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:00:08.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:00:08.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:456: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2026-02-13T02:00:08.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:457: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2026-02-13T02:00:08.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:00:08.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:00:08.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-13T02:00:08.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:00:08.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:00:08.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:00:08.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:00:08.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:00:08.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:00:08.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:00:08.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:00:08.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:00:08.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:00:08.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:00:08.976 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:00:08.976 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:00:08.976 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:00:08.976 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:00:08.977 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:00:08.977 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:00:08.977 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:00:08.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:00:08.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:00:08.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:00:08.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:00:08.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:00:08.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:00:08.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:00:08.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:00:08.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:00:08.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:00:08.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:00:08.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2026-02-13T02:00:08.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:00:08.978 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:00:08.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3fedb813-271a-461f-b9b9-f4b52784004a 2026-02-13T02:00:08.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 3fedb813-271a-461f-b9b9-f4b52784004a' 2026-02-13T02:00:08.979 INFO:tasks.workunit.client.0.trial112.stdout:add osd1 3fedb813-271a-461f-b9b9-f4b52784004a 2026-02-13T02:00:08.979 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:00:08.988 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCohY5pP/vAOhAADkC7NVMBfX0Eh4AITVT7UQ== 2026-02-13T02:00:08.988 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCohY5pP/vAOhAADkC7NVMBfX0Eh4AITVT7UQ=="}' 2026-02-13T02:00:08.988 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3fedb813-271a-461f-b9b9-f4b52784004a -i td/osd-scrub-repair/1/new.json 2026-02-13T02:00:09.080 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:00:09.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-13T02:00:09.089 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQCohY5pP/vAOhAADkC7NVMBfX0Eh4AITVT7UQ== --osd-uuid 3fedb813-271a-461f-b9b9-f4b52784004a 2026-02-13T02:00:09.107 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:09.102+0000 7f2f19219a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:09.108 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:09.106+0000 7f2f19219a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:09.109 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:09.106+0000 7f2f19219a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:09.110 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:09.106+0000 7f2f19219a40 -1 bdev(0x560fa4c12c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-13T02:00:09.110 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:09.106+0000 7f2f19219a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-13T02:00:09.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-13T02:00:09.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:00:09.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-13T02:00:09.757 INFO:tasks.workunit.client.0.trial112.stdout:adding osd1 key to auth repository 2026-02-13T02:00:09.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:00:09.957 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T02:00:09.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-13T02:00:09.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2026-02-13T02:00:09.959 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:00:09.959 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:00:09.959 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:00:09.982 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:09.978+0000 7f752eee0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:09.984 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:09.978+0000 7f752eee0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:09.985 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:09.982+0000 7f752eee0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:10.123 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-13T02:00:10.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:00:10.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:00:10.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:00:10.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:00:10.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:00:10.124 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:00:10.125 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:00:10.125 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:00:10.125 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:00:10.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:00:10.370 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:10.366+0000 7f752eee0a40 -1 Falling back to public interface 2026-02-13T02:00:10.992 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:10.986+0000 7f752eee0a40 -1 osd.1 0 log_to_monitors true 2026-02-13T02:00:11.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:00:11.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:00:11.284 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:00:11.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:00:11.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:00:11.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:00:11.440 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/229621484,v1:127.0.0.1:6811/229621484] [v2:127.0.0.1:6812/229621484,v1:127.0.0.1:6813/229621484] exists,up 3fedb813-271a-461f-b9b9-f4b52784004a 2026-02-13T02:00:11.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:00:11.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:00:11.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:00:11.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:456: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2026-02-13T02:00:11.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:457: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2026-02-13T02:00:11.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:00:11.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:00:11.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-13T02:00:11.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:00:11.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-13T02:00:11.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:00:11.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:00:11.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:00:11.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:00:11.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-13T02:00:11.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-13T02:00:11.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:00:11.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:00:11.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:00:11.445 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:00:11.445 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:00:11.445 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:00:11.445 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:00:11.445 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:00:11.445 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:00:11.445 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:00:11.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:00:11.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:00:11.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:00:11.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:00:11.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:00:11.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:00:11.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:00:11.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:00:11.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:00:11.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:00:11.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:00:11.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2026-02-13T02:00:11.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-13T02:00:11.447 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:00:11.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=801087c9-2833-42d9-beb1-b36eefdc1497 2026-02-13T02:00:11.447 INFO:tasks.workunit.client.0.trial112.stdout:add osd2 801087c9-2833-42d9-beb1-b36eefdc1497 2026-02-13T02:00:11.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 801087c9-2833-42d9-beb1-b36eefdc1497' 2026-02-13T02:00:11.448 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:00:11.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCrhY5pdygvGxAAKNN+l9VCI1Xw4zDBxyyAwg== 2026-02-13T02:00:11.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCrhY5pdygvGxAAKNN+l9VCI1Xw4zDBxyyAwg=="}' 2026-02-13T02:00:11.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 801087c9-2833-42d9-beb1-b36eefdc1497 -i td/osd-scrub-repair/2/new.json 2026-02-13T02:00:11.629 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:00:11.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-13T02:00:11.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQCrhY5pdygvGxAAKNN+l9VCI1Xw4zDBxyyAwg== --osd-uuid 801087c9-2833-42d9-beb1-b36eefdc1497 2026-02-13T02:00:11.659 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:11.654+0000 7f73a1156a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:11.661 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:11.658+0000 7f73a1156a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:11.662 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:11.658+0000 7f73a1156a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:11.662 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:11.658+0000 7f73a1156a40 -1 bdev(0x55b2110fcc00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-13T02:00:11.663 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:11.658+0000 7f73a1156a40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-13T02:00:12.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-13T02:00:12.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:00:12.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-13T02:00:12.297 INFO:tasks.workunit.client.0.trial112.stdout:adding osd2 key to auth repository 2026-02-13T02:00:12.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:00:12.498 INFO:tasks.workunit.client.0.trial112.stdout:start osd.2 2026-02-13T02:00:12.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-13T02:00:12.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2026-02-13T02:00:12.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:00:12.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:00:12.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:00:12.524 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:12.518+0000 7fc41c81ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:12.526 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:12.522+0000 7fc41c81ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:12.527 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:12.522+0000 7fc41c81ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:12.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-13T02:00:12.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:00:12.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T02:00:12.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:00:12.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:00:12.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:00:12.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:00:12.658 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:00:12.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:00:12.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:00:12.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:00:12.942 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:12.938+0000 7fc41c81ba40 -1 Falling back to public interface 2026-02-13T02:00:13.538 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:13.534+0000 7fc41c81ba40 -1 osd.2 0 log_to_monitors true 2026-02-13T02:00:13.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:00:13.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:00:13.815 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:00:13.816 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:00:13.816 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:00:13.816 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:00:13.967 INFO:tasks.workunit.client.0.trial112.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4122702890,v1:127.0.0.1:6819/4122702890] [v2:127.0.0.1:6820/4122702890,v1:127.0.0.1:6821/4122702890] exists,up 801087c9-2833-42d9-beb1-b36eefdc1497 2026-02-13T02:00:13.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:00:13.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:00:13.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:00:13.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_tag: create_pool testpool 1 1 2026-02-13T02:00:13.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create testpool 1 1 2026-02-13T02:00:14.176 INFO:tasks.workunit.client.0.trial112.stderr:pool 'testpool' created 2026-02-13T02:00:14.190 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:00:15.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:461: TEST_auto_repair_bluestore_tag: ceph osd pool set testpool size 2 2026-02-13T02:00:15.394 INFO:tasks.workunit.client.0.trial112.stderr:set pool 1 size to 2 2026-02-13T02:00:15.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:462: TEST_auto_repair_bluestore_tag: wait_for_clean 2026-02-13T02:00:15.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:00:15.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:00:15.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:00:15.408 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:00:15.408 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:00:15.408 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:00:15.408 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:00:15.409 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:00:15.409 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:00:15.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:00:15.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:00:15.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:00:15.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:00:15.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:00:15.517 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:00:15.681 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:00:15.681 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:00:15.682 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:00:15.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:00:15.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:00:15.682 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:00:15.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-13T02:00:15.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-13T02:00:15.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-13T02:00:15.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:00:15.756 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:00:15.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705666 2026-02-13T02:00:15.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705666 2026-02-13T02:00:15.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705666' 2026-02-13T02:00:15.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:00:15.829 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:00:15.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542146 2026-02-13T02:00:15.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542146 2026-02-13T02:00:15.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705666 2-60129542146' 2026-02-13T02:00:15.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:00:15.900 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-13T02:00:15.901 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:00:15.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:00:15.903 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-13T02:00:15.903 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:00:15.904 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-13T02:00:15.904 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836483 2026-02-13T02:00:15.905 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-13T02:00:15.905 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:00:16.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-13T02:00:16.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:00:17.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:00:17.069 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:00:17.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-13T02:00:17.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:00:17.226 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705666 2026-02-13T02:00:17.227 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:00:17.228 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:00:17.228 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705666 2026-02-13T02:00:17.229 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:00:17.230 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705666 2026-02-13T02:00:17.230 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705666 2026-02-13T02:00:17.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705666' 2026-02-13T02:00:17.231 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:00:17.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705666 -lt 38654705666 2026-02-13T02:00:17.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:00:17.400 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542146 2026-02-13T02:00:17.400 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:00:17.401 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:00:17.402 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542146 2026-02-13T02:00:17.402 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:00:17.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542146 2026-02-13T02:00:17.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542146' 2026-02-13T02:00:17.404 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542146 2026-02-13T02:00:17.405 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:00:17.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542146 -lt 60129542146 2026-02-13T02:00:17.580 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:00:17.580 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:00:17.581 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:00:17.779 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:00:17.779 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:00:17.780 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:00:17.781 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:00:17.781 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:00:17.781 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:00:17.781 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:00:17.781 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:00:17.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:00:17.937 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:00:17.937 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:00:17.938 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:00:18.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:00:18.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:00:18.128 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:00:18.128 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:465: TEST_auto_repair_bluestore_tag: local payload=ABCDEF 2026-02-13T02:00:18.128 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:466: TEST_auto_repair_bluestore_tag: echo ABCDEF 2026-02-13T02:00:18.128 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:467: TEST_auto_repair_bluestore_tag: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-13T02:00:18.147 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:471: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2026-02-13T02:00:18.147 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-13T02:00:18.147 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-13T02:00:18.148 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2026-02-13T02:00:18.148 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-13T02:00:18.149 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-13T02:00:18.149 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-13T02:00:18.149 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T02:00:18.308 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-13T02:00:18.309 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2026-02-13T02:00:18.309 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-13T02:00:18.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:471: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2026-02-13T02:00:18.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:00:18.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:00:18.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T02:00:18.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:00:18.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2026-02-13T02:00:18.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:00:18.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:00:18.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T02:00:18.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:00:18.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T02:00:18.478 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:00:18.478 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:00:18.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:00:18.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:00:18.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:00:18.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:00:18.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2026-02-13T02:00:18.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:00:18.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:00:18.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:00:18.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:00:18.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:00:18.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2026-02-13T02:00:18.958 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:eb822e21:::SOMETHING:head# 2026-02-13T02:00:19.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2026-02-13T02:00:19.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:00:19.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:00:19.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:00:19.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:00:19.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:00:19.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:00:19.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:00:19.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:00:19.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:00:19.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:00:19.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:00:19.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:00:19.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:00:19.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:00:19.060 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:00:19.060 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:00:19.061 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:00:19.061 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:00:19.061 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:00:19.061 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:00:19.061 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:00:19.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:00:19.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:00:19.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:00:19.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:00:19.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:00:19.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:00:19.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:00:19.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:00:19.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:00:19.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2026-02-13T02:00:19.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:00:19.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:00:19.063 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T02:00:19.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2026-02-13T02:00:19.063 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T02:00:19.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:00:19.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:00:19.064 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:00:19.064 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:00:19.081 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:19.078+0000 7fa5a9adea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:19.082 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:19.078+0000 7fa5a9adea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:19.083 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:19.078+0000 7fa5a9adea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:19.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:00:19.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:00:19.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:00:19.222 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:00:19.222 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:00:19.222 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:00:19.222 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:00:19.222 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:00:19.222 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:00:19.222 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:00:19.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:00:19.430 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:19.426+0000 7fa5a9adea40 -1 Falling back to public interface 2026-02-13T02:00:20.082 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:20.078+0000 7fa5a9adea40 -1 osd.0 20 log_to_monitors true 2026-02-13T02:00:20.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:00:20.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:00:20.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:00:20.390 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:00:20.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:00:20.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:00:20.551 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 21 last_clean_interval [5,20) [v2:127.0.0.1:6800/2767404006,v1:127.0.0.1:6801/2767404006] [v2:127.0.0.1:6802/2767404006,v1:127.0.0.1:6803/2767404006] exists,up 39d03e9f-959e-4ffa-b825-1766b89b0da8 2026-02-13T02:00:20.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:00:20.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:00:20.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:00:20.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:00:20.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:00:20.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:00:20.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:00:20.552 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:00:20.553 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:00:20.553 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:00:20.553 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:00:20.553 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:00:20.553 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:00:20.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:00:20.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:00:20.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:00:20.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:00:20.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:00:20.653 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:00:20.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:00:20.815 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:00:20.815 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:00:20.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:00:20.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:00:20.815 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:00:20.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215106 2026-02-13T02:00:20.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215106 2026-02-13T02:00:20.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215106' 2026-02-13T02:00:20.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:00:20.888 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:00:20.959 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705668 2026-02-13T02:00:20.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705668 2026-02-13T02:00:20.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215106 1-38654705668' 2026-02-13T02:00:20.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:00:20.960 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:00:21.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542148 2026-02-13T02:00:21.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542148 2026-02-13T02:00:21.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215106 1-38654705668 2-60129542148' 2026-02-13T02:00:21.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:00:21.041 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-103079215106 2026-02-13T02:00:21.041 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:00:21.042 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:00:21.043 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-103079215106 2026-02-13T02:00:21.043 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:00:21.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215106 2026-02-13T02:00:21.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2026-02-13T02:00:21.045 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 103079215106 2026-02-13T02:00:21.045 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:00:21.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215106 -lt 103079215106 2026-02-13T02:00:21.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:00:21.209 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705668 2026-02-13T02:00:21.209 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:00:21.210 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:00:21.211 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705668 2026-02-13T02:00:21.211 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:00:21.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705668 2026-02-13T02:00:21.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705668' 2026-02-13T02:00:21.212 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 38654705668 2026-02-13T02:00:21.213 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:00:21.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705668 -lt 38654705668 2026-02-13T02:00:21.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:00:21.375 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542148 2026-02-13T02:00:21.375 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:00:21.376 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:00:21.377 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542148 2026-02-13T02:00:21.377 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:00:21.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542148 2026-02-13T02:00:21.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542148' 2026-02-13T02:00:21.378 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 60129542148 2026-02-13T02:00:21.379 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:00:21.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542147 -lt 60129542148 2026-02-13T02:00:21.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:00:22.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:00:22.545 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:00:22.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542147 -lt 60129542148 2026-02-13T02:00:22.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:00:23.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:00:23.712 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:00:23.876 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542148 -lt 60129542148 2026-02-13T02:00:23.877 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:00:23.877 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:00:23.877 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:00:24.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:00:24.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:00:24.074 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:00:24.074 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:00:24.074 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:00:24.074 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:00:24.074 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:00:24.074 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:00:24.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:00:24.235 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:00:24.236 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:00:24.236 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:00:24.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:00:24.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:00:24.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:00:24.430 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:473: TEST_auto_repair_bluestore_tag: get_pg testpool SOMETHING 2026-02-13T02:00:24.430 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2026-02-13T02:00:24.430 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-13T02:00:24.431 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool SOMETHING 2026-02-13T02:00:24.431 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T02:00:24.594 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:473: TEST_auto_repair_bluestore_tag: local pgid=1.0 2026-02-13T02:00:24.595 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:474: TEST_auto_repair_bluestore_tag: get_primary testpool SOMETHING 2026-02-13T02:00:24.595 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-13T02:00:24.595 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-13T02:00:24.595 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-13T02:00:24.595 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T02:00:24.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:474: TEST_auto_repair_bluestore_tag: local primary=1 2026-02-13T02:00:24.758 INFO:tasks.workunit.client.0.trial112.stdout:Affected PG 1.0 w/ primary 1 2026-02-13T02:00:24.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:475: TEST_auto_repair_bluestore_tag: echo 'Affected PG ' 1.0 ' w/ primary ' 1 2026-02-13T02:00:24.759 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_tag: get_last_scrub_stamp 1.0 2026-02-13T02:00:24.759 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:00:24.759 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:00:24.759 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:00:24.759 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:00:24.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_tag: local last_scrub_stamp=2026-02-13T02:00:14.173958+0000 2026-02-13T02:00:24.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:477: TEST_auto_repair_bluestore_tag: initiate_and_fetch_state 1 1.0 3.0 2026-02-13T02:00:24.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:382: initiate_and_fetch_state: local the_osd=osd.1 2026-02-13T02:00:24.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:383: initiate_and_fetch_state: local pgid=1.0 2026-02-13T02:00:24.917 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:384: initiate_and_fetch_state: get_last_scrub_stamp 1.0 2026-02-13T02:00:24.917 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:00:24.917 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:00:24.917 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:00:24.917 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:00:25.072 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:384: initiate_and_fetch_state: local last_scrub=2026-02-13T02:00:14.173958+0000 2026-02-13T02:00:25.072 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:386: initiate_and_fetch_state: set_config osd 1 osd_scrub_sleep 3.0 2026-02-13T02:00:25.072 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-13T02:00:25.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2026-02-13T02:00:25.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=osd_scrub_sleep 2026-02-13T02:00:25.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=3.0 2026-02-13T02:00:25.074 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-13T02:00:25.074 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2026-02-13T02:00:25.074 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-13T02:00:25.074 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-13T02:00:25.074 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:00:25.075 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:00:25.075 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:00:25.075 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-osd.1.asok 2026-02-13T02:00:25.075 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.35207/ceph-osd.1.asok config set osd_scrub_sleep 3.0 2026-02-13T02:00:25.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-13T02:00:25.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:387: initiate_and_fetch_state: set_config osd 1 osd_scrub_auto_repair true 2026-02-13T02:00:25.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-13T02:00:25.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2026-02-13T02:00:25.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=osd_scrub_auto_repair 2026-02-13T02:00:25.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-13T02:00:25.135 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-13T02:00:25.135 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2026-02-13T02:00:25.136 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-13T02:00:25.136 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-13T02:00:25.136 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:00:25.136 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:00:25.136 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:00:25.136 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-osd.1.asok 2026-02-13T02:00:25.136 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.35207/ceph-osd.1.asok config set osd_scrub_auto_repair true 2026-02-13T02:00:25.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-13T02:00:25.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:389: initiate_and_fetch_state: flush_pg_stats 2026-02-13T02:00:25.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:00:25.193 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:00:25.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:00:25.356 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:00:25.356 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:00:25.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:00:25.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:00:25.356 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:00:25.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215108 2026-02-13T02:00:25.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215108 2026-02-13T02:00:25.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215108' 2026-02-13T02:00:25.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:00:25.429 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:00:25.502 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705670 2026-02-13T02:00:25.502 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705670 2026-02-13T02:00:25.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215108 1-38654705670' 2026-02-13T02:00:25.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:00:25.503 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:00:25.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542150 2026-02-13T02:00:25.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542150 2026-02-13T02:00:25.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215108 1-38654705670 2-60129542150' 2026-02-13T02:00:25.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:00:25.578 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-103079215108 2026-02-13T02:00:25.579 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:00:25.580 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:00:25.581 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-103079215108 2026-02-13T02:00:25.581 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:00:25.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215108 2026-02-13T02:00:25.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 103079215108' 2026-02-13T02:00:25.583 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 103079215108 2026-02-13T02:00:25.583 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:00:25.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215106 -lt 103079215108 2026-02-13T02:00:25.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:00:26.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:00:26.753 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:00:26.927 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215106 -lt 103079215108 2026-02-13T02:00:26.927 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:00:27.929 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:00:27.929 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:00:28.089 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215108 -lt 103079215108 2026-02-13T02:00:28.089 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:00:28.090 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705670 2026-02-13T02:00:28.090 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:00:28.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:00:28.092 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705670 2026-02-13T02:00:28.092 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:00:28.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705670 2026-02-13T02:00:28.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705670' 2026-02-13T02:00:28.094 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705670 2026-02-13T02:00:28.094 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:00:28.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705670 -lt 38654705670 2026-02-13T02:00:28.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:00:28.266 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542150 2026-02-13T02:00:28.266 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:00:28.267 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:00:28.268 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542150 2026-02-13T02:00:28.268 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:00:28.270 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542150 2026-02-13T02:00:28.270 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542150' 2026-02-13T02:00:28.270 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542150 2026-02-13T02:00:28.270 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:00:28.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542150 -lt 60129542150 2026-02-13T02:00:28.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:390: initiate_and_fetch_state: date --rfc-3339=ns 2026-02-13T02:00:28.435 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13 02:00:28.434210845+00:00 2026-02-13T02:00:28.436 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:393: initiate_and_fetch_state: get_asok_path osd.1 2026-02-13T02:00:28.436 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-13T02:00:28.436 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-13T02:00:28.437 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:00:28.437 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:00:28.437 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:00:28.437 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-osd.1.asok 2026-02-13T02:00:28.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:393: initiate_and_fetch_state: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.35207/ceph-osd.1.asok schedule-deep-scrub 1.0 2026-02-13T02:00:28.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i=0 )) 2026-02-13T02:00:28.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i < 80 )) 2026-02-13T02:00:28.504 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: ceph pg 1.0 query --format json 2026-02-13T02:00:28.504 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: jq .state 2026-02-13T02:00:28.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: st='"active+clean"' 2026-02-13T02:00:28.578 INFO:tasks.workunit.client.0.trial112.stdout:{"deep":true,"must":false,"stamp":"2026-02-01T21:12:18.495728+0000"}0 ) state now: "active+clean" 2026-02-13T02:00:28.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:399: initiate_and_fetch_state: echo 0 ') state now: ' '"active+clean"' 2026-02-13T02:00:28.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: initiate_and_fetch_state: case "$st" in 2026-02-13T02:00:28.579 INFO:tasks.workunit.client.0.trial112.stdout:"active+clean" 2026-02-13T02:00:28.580 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: initiate_and_fetch_state: echo '"active+clean"' 2026-02-13T02:00:28.580 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:409: initiate_and_fetch_state: '[' 0 == 4 ']' 2026-02-13T02:00:28.580 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:412: initiate_and_fetch_state: sleep 0.3 2026-02-13T02:00:28.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i++ )) 2026-02-13T02:00:28.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i < 80 )) 2026-02-13T02:00:28.881 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: ceph pg 1.0 query --format json 2026-02-13T02:00:28.881 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: jq .state 2026-02-13T02:00:28.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: st='"active+clean+scrubbing+deep"' 2026-02-13T02:00:28.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:399: initiate_and_fetch_state: echo 1 ') state now: ' '"active+clean+scrubbing+deep"' 2026-02-13T02:00:28.953 INFO:tasks.workunit.client.0.trial112.stdout:1 ) state now: "active+clean+scrubbing+deep" 2026-02-13T02:00:28.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: initiate_and_fetch_state: case "$st" in 2026-02-13T02:00:28.954 INFO:tasks.workunit.client.0.trial112.stdout:found scrub 2026-02-13T02:00:28.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:403: initiate_and_fetch_state: echo 'found scrub' 2026-02-13T02:00:28.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:403: initiate_and_fetch_state: return 0 2026-02-13T02:00:28.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:478: TEST_auto_repair_bluestore_tag: r=0 2026-02-13T02:00:28.955 INFO:tasks.workunit.client.0.trial112.stdout:initiate_and_fetch_state ret: 0 2026-02-13T02:00:28.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:479: TEST_auto_repair_bluestore_tag: echo 'initiate_and_fetch_state ret: ' 0 2026-02-13T02:00:28.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:480: TEST_auto_repair_bluestore_tag: set_config osd td/osd-scrub-repair osd_scrub_sleep 0 2026-02-13T02:00:28.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-13T02:00:28.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=td/osd-scrub-repair 2026-02-13T02:00:28.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=osd_scrub_sleep 2026-02-13T02:00:28.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=0 2026-02-13T02:00:28.957 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-13T02:00:28.957 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.td/osd-scrub-repair 2026-02-13T02:00:28.957 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.td/osd-scrub-repair 2026-02-13T02:00:28.957 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.td/osd-scrub-repair ']' 2026-02-13T02:00:28.957 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:00:28.957 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:00:28.957 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:00:28.957 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-osd.td/osd-scrub-repair.asok 2026-02-13T02:00:28.957 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.35207/ceph-osd.td/osd-scrub-repair.asok config set osd_scrub_sleep 0 2026-02-13T02:00:29.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test == true 2026-02-13T02:00:29.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 1159: test: ==: unary operator expected 2026-02-13T02:00:29.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:481: TEST_auto_repair_bluestore_tag: '[' 0 -ne 0 ']' 2026-02-13T02:00:29.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:485: TEST_auto_repair_bluestore_tag: wait_end_of_scrub 1 1.0 2026-02-13T02:00:29.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:420: wait_end_of_scrub: local the_osd=osd.1 2026-02-13T02:00:29.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:421: wait_end_of_scrub: local pgid=1.0 2026-02-13T02:00:29.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: wait_end_of_scrub: (( i=0 )) 2026-02-13T02:00:29.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: wait_end_of_scrub: (( i < 40 )) 2026-02-13T02:00:29.003 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: ceph pg 1.0 query --format json 2026-02-13T02:00:29.003 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: jq .state 2026-02-13T02:00:29.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2026-02-13T02:00:29.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:425: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2026-02-13T02:00:29.077 INFO:tasks.workunit.client.0.trial112.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2026-02-13T02:00:29.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:426: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2026-02-13T02:00:29.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:427: wait_end_of_scrub: '[' 0 == 4 ']' 2026-02-13T02:00:29.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: wait_end_of_scrub: sleep 0.3 2026-02-13T02:00:29.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: wait_end_of_scrub: (( i++ )) 2026-02-13T02:00:29.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: wait_end_of_scrub: (( i < 40 )) 2026-02-13T02:00:29.379 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: ceph pg 1.0 query --format json 2026-02-13T02:00:29.379 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: jq .state 2026-02-13T02:00:29.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2026-02-13T02:00:29.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:425: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2026-02-13T02:00:29.454 INFO:tasks.workunit.client.0.trial112.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2026-02-13T02:00:29.455 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:426: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2026-02-13T02:00:29.455 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:427: wait_end_of_scrub: '[' 1 == 4 ']' 2026-02-13T02:00:29.455 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: wait_end_of_scrub: sleep 0.3 2026-02-13T02:00:29.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: wait_end_of_scrub: (( i++ )) 2026-02-13T02:00:29.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: wait_end_of_scrub: (( i < 40 )) 2026-02-13T02:00:29.756 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: jq .state 2026-02-13T02:00:29.756 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: ceph pg 1.0 query --format json 2026-02-13T02:00:29.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2026-02-13T02:00:29.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:425: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2026-02-13T02:00:29.830 INFO:tasks.workunit.client.0.trial112.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2026-02-13T02:00:29.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:426: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2026-02-13T02:00:29.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:427: wait_end_of_scrub: '[' 2 == 4 ']' 2026-02-13T02:00:29.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: wait_end_of_scrub: sleep 0.3 2026-02-13T02:00:30.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: wait_end_of_scrub: (( i++ )) 2026-02-13T02:00:30.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: wait_end_of_scrub: (( i < 40 )) 2026-02-13T02:00:30.132 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: ceph pg 1.0 query --format json 2026-02-13T02:00:30.132 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: jq .state 2026-02-13T02:00:30.206 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2026-02-13T02:00:30.206 INFO:tasks.workunit.client.0.trial112.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2026-02-13T02:00:30.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:425: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2026-02-13T02:00:30.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:426: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2026-02-13T02:00:30.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:427: wait_end_of_scrub: '[' 3 == 4 ']' 2026-02-13T02:00:30.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: wait_end_of_scrub: sleep 0.3 2026-02-13T02:00:30.507 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: wait_end_of_scrub: (( i++ )) 2026-02-13T02:00:30.507 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: wait_end_of_scrub: (( i < 40 )) 2026-02-13T02:00:30.508 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: ceph pg 1.0 query --format json 2026-02-13T02:00:30.509 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: jq .state 2026-02-13T02:00:30.581 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2026-02-13T02:00:30.582 INFO:tasks.workunit.client.0.trial112.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2026-02-13T02:00:30.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:425: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2026-02-13T02:00:30.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:426: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2026-02-13T02:00:30.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:427: wait_end_of_scrub: '[' 4 == 4 ']' 2026-02-13T02:00:30.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:428: wait_end_of_scrub: flush_pg_stats 2026-02-13T02:00:30.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:00:30.583 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:00:30.739 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:00:30.739 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:00:30.739 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:00:30.739 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:00:30.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:00:30.740 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:00:30.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215110 2026-02-13T02:00:30.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215110 2026-02-13T02:00:30.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215110' 2026-02-13T02:00:30.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:00:30.813 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:00:30.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705672 2026-02-13T02:00:30.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705672 2026-02-13T02:00:30.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215110 1-38654705672' 2026-02-13T02:00:30.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:00:30.888 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:00:30.959 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542152 2026-02-13T02:00:30.959 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542152 2026-02-13T02:00:30.959 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215110 1-38654705672 2-60129542152' 2026-02-13T02:00:30.959 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:00:30.960 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-103079215110 2026-02-13T02:00:30.960 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:00:30.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:00:30.961 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-103079215110 2026-02-13T02:00:30.961 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:00:30.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215110 2026-02-13T02:00:30.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 103079215110' 2026-02-13T02:00:30.963 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 103079215110 2026-02-13T02:00:30.964 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:00:31.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215110 -lt 103079215110 2026-02-13T02:00:31.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:00:31.136 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705672 2026-02-13T02:00:31.136 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:00:31.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:00:31.138 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705672 2026-02-13T02:00:31.138 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:00:31.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705672 2026-02-13T02:00:31.140 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705672 2026-02-13T02:00:31.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705672' 2026-02-13T02:00:31.140 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:00:31.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705672 -lt 38654705672 2026-02-13T02:00:31.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:00:31.309 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542152 2026-02-13T02:00:31.309 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:00:31.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:00:31.311 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542152 2026-02-13T02:00:31.311 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:00:31.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542152 2026-02-13T02:00:31.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542152' 2026-02-13T02:00:31.313 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542152 2026-02-13T02:00:31.314 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:00:31.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542152 -lt 60129542152 2026-02-13T02:00:31.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: wait_end_of_scrub: sleep 0.3 2026-02-13T02:00:31.779 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: wait_end_of_scrub: (( i++ )) 2026-02-13T02:00:31.779 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: wait_end_of_scrub: (( i < 40 )) 2026-02-13T02:00:31.780 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: ceph pg 1.0 query --format json 2026-02-13T02:00:31.780 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: jq .state 2026-02-13T02:00:31.857 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2026-02-13T02:00:31.857 INFO:tasks.workunit.client.0.trial112.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2026-02-13T02:00:31.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:425: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2026-02-13T02:00:31.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:426: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2026-02-13T02:00:31.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:427: wait_end_of_scrub: '[' 0 == 4 ']' 2026-02-13T02:00:31.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: wait_end_of_scrub: sleep 0.3 2026-02-13T02:00:32.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: wait_end_of_scrub: (( i++ )) 2026-02-13T02:00:32.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: wait_end_of_scrub: (( i < 40 )) 2026-02-13T02:00:32.160 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: ceph pg 1.0 query --format json 2026-02-13T02:00:32.160 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: jq .state 2026-02-13T02:00:32.233 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: st='"active+clean"' 2026-02-13T02:00:32.233 INFO:tasks.workunit.client.0.trial112.stdout:wait-scrub-end state now: "active+clean" 2026-02-13T02:00:32.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:425: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean"' 2026-02-13T02:00:32.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:426: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2026-02-13T02:00:32.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:426: wait_end_of_scrub: break 2026-02-13T02:00:32.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:433: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2026-02-13T02:00:32.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:438: wait_end_of_scrub: return 0 2026-02-13T02:00:32.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:486: TEST_auto_repair_bluestore_tag: ceph pg dump pgs 2026-02-13T02:00:32.380 INFO:tasks.workunit.client.0.trial112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-13T02:00:32.380 INFO:tasks.workunit.client.0.trial112.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean+scrubbing+deep 2026-02-13T02:00:28.891571+0000 20'1 26:48 [1,0] 1 [1,0] 1 0'0 2026-02-13T02:00:14.173958+0000 0'0 2026-02-01T21:12:18.495728+0000 0 0 deep scrubbing for 2s 0 0 2026-02-13T02:00:32.380 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T02:00:32.380 INFO:tasks.workunit.client.0.trial112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-13T02:00:32.380 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:00:32.389 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:490: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2026-02-13T02:00:32.389 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-13T02:00:32.389 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-13T02:00:32.390 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2026-02-13T02:00:32.390 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-13T02:00:32.390 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-13T02:00:32.391 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-13T02:00:32.391 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T02:00:32.549 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-13T02:00:32.550 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2026-02-13T02:00:32.550 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-13T02:00:32.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:490: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-13T02:00:32.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:00:32.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:00:32.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T02:00:32.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:00:32.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-13T02:00:32.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:00:32.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:00:32.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T02:00:32.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:00:32.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T02:00:32.710 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:00:32.710 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:00:32.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:00:32.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:00:32.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:00:33.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:00:33.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-13T02:00:33.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:00:33.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:00:33.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:00:33.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:00:33.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:00:33.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2026-02-13T02:00:33.078 INFO:tasks.workunit.client.0.trial112.stdout:_ 2026-02-13T02:00:33.078 INFO:tasks.workunit.client.0.trial112.stdout:snapset 2026-02-13T02:00:33.182 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2026-02-13T02:00:33.182 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:00:33.182 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:00:33.182 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:00:33.182 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:00:33.182 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:00:33.182 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:00:33.182 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:00:33.182 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:00:33.183 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:00:33.183 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:00:33.183 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:00:33.183 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:00:33.183 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:00:33.183 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:00:33.183 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:00:33.183 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:00:33.183 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:00:33.183 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:00:33.183 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:00:33.183 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:00:33.184 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:00:33.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:00:33.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:00:33.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:00:33.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:00:33.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:00:33.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:00:33.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:00:33.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:00:33.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:00:33.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2026-02-13T02:00:33.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:00:33.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:00:33.186 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T02:00:33.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2026-02-13T02:00:33.187 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T02:00:33.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:00:33.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:00:33.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:00:33.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:00:33.200 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:33.193+0000 7fb514173a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:33.202 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:33.197+0000 7fb514173a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:33.203 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:33.197+0000 7fb514173a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:33.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:00:33.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:00:33.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:00:33.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:00:33.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:00:33.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:00:33.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:00:33.351 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:00:33.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:00:33.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:00:33.512 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:00:33.590 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:33.585+0000 7fb514173a40 -1 Falling back to public interface 2026-02-13T02:00:34.183 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:34.177+0000 7fb514173a40 -1 osd.0 26 log_to_monitors true 2026-02-13T02:00:34.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:00:34.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:00:34.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:00:34.514 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:00:34.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:00:34.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:00:34.673 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [24,26) [v2:127.0.0.1:6800/3273300567,v1:127.0.0.1:6801/3273300567] [v2:127.0.0.1:6802/3273300567,v1:127.0.0.1:6803/3273300567] exists,up 39d03e9f-959e-4ffa-b825-1766b89b0da8 2026-02-13T02:00:34.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:00:34.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:00:34.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:00:34.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:00:34.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:00:34.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:00:34.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:00:34.674 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:00:34.675 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:00:34.675 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:00:34.675 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:00:34.675 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:00:34.675 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:00:34.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:00:34.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:00:34.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:00:34.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:00:34.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:00:34.781 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:00:34.944 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:00:34.944 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:00:34.944 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:00:34.944 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:00:34.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:00:34.945 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:00:35.024 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018882 2026-02-13T02:00:35.024 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018882 2026-02-13T02:00:35.024 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018882' 2026-02-13T02:00:35.024 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:00:35.024 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:00:35.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705674 2026-02-13T02:00:35.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705674 2026-02-13T02:00:35.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018882 1-38654705674' 2026-02-13T02:00:35.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:00:35.096 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:00:35.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542154 2026-02-13T02:00:35.167 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542154 2026-02-13T02:00:35.167 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018882 1-38654705674 2-60129542154' 2026-02-13T02:00:35.167 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:00:35.167 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-128849018882 2026-02-13T02:00:35.168 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:00:35.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:00:35.169 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-128849018882 2026-02-13T02:00:35.170 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:00:35.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018882 2026-02-13T02:00:35.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2026-02-13T02:00:35.171 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 128849018882 2026-02-13T02:00:35.172 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:00:35.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 128849018882 2026-02-13T02:00:35.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:00:36.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:00:36.347 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:00:36.508 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 128849018882 2026-02-13T02:00:36.508 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:00:37.510 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:00:37.510 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:00:37.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018882 -lt 128849018882 2026-02-13T02:00:37.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:00:37.675 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705674 2026-02-13T02:00:37.675 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:00:37.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:00:37.677 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705674 2026-02-13T02:00:37.677 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:00:37.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705674 2026-02-13T02:00:37.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705674' 2026-02-13T02:00:37.679 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 38654705674 2026-02-13T02:00:37.679 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:00:37.847 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705674 -lt 38654705674 2026-02-13T02:00:37.847 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:00:37.847 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542154 2026-02-13T02:00:37.848 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:00:37.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:00:37.849 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542154 2026-02-13T02:00:37.850 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:00:37.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542154 2026-02-13T02:00:37.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542154' 2026-02-13T02:00:37.851 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 60129542154 2026-02-13T02:00:37.852 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:00:38.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542154 -lt 60129542154 2026-02-13T02:00:38.013 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:00:38.014 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:00:38.014 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:00:38.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:00:38.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:00:38.206 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:00:38.206 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:00:38.206 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:00:38.207 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:00:38.207 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:00:38.207 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:00:38.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:00:38.365 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:00:38.365 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:00:38.365 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:00:38.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:00:38.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:00:38.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:00:38.555 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:491: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2026-02-13T02:00:38.555 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-13T02:00:38.555 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-13T02:00:38.556 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2026-02-13T02:00:38.556 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-13T02:00:38.556 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-13T02:00:38.557 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-13T02:00:38.557 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T02:00:38.724 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-13T02:00:38.725 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2026-02-13T02:00:38.725 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-13T02:00:38.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:491: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2026-02-13T02:00:38.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:00:38.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:00:38.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T02:00:38.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:00:38.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2026-02-13T02:00:38.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:00:38.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:00:38.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T02:00:38.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:00:38.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T02:00:38.888 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:00:38.888 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:00:38.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:00:38.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:00:38.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:00:39.196 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:00:39.196 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2026-02-13T02:00:39.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:00:39.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:00:39.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:00:39.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:00:39.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:00:39.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2026-02-13T02:00:39.383 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2026-02-13T02:00:39.383 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:00:39.383 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:00:39.383 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:00:39.383 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:00:39.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:00:39.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:00:39.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:00:39.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:00:39.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:00:39.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:00:39.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:00:39.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:00:39.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:00:39.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:00:39.384 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:00:39.385 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:00:39.385 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:00:39.385 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:00:39.385 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:00:39.385 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:00:39.385 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:00:39.385 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:00:39.385 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:00:39.385 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:00:39.385 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:00:39.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:00:39.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:00:39.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:00:39.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:00:39.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:00:39.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2026-02-13T02:00:39.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:00:39.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:00:39.386 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T02:00:39.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2026-02-13T02:00:39.388 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T02:00:39.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:00:39.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:00:39.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:00:39.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:00:39.402 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:39.397+0000 7fbfd12afa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:39.404 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:39.397+0000 7fbfd12afa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:39.405 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:39.401+0000 7fbfd12afa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:39.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:00:39.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:00:39.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:00:39.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:00:39.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:00:39.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:00:39.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:00:39.541 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:00:39.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:00:39.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:00:39.704 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:00:39.750 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:39.745+0000 7fbfd12afa40 -1 Falling back to public interface 2026-02-13T02:00:40.406 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:40.401+0000 7fbfd12afa40 -1 osd.0 31 log_to_monitors true 2026-02-13T02:00:40.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:00:40.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:00:40.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:00:40.706 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:00:40.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:00:40.707 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:00:40.868 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 35 up_thru 0 down_at 32 last_clean_interval [30,31) [v2:127.0.0.1:6800/3394416661,v1:127.0.0.1:6801/3394416661] [v2:127.0.0.1:6802/3394416661,v1:127.0.0.1:6803/3394416661] exists,up 39d03e9f-959e-4ffa-b825-1766b89b0da8 2026-02-13T02:00:40.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:00:40.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:00:40.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:00:40.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:00:40.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:00:40.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:00:40.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:00:40.869 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:00:40.870 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:00:40.870 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:00:40.870 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:00:40.870 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:00:40.870 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:00:40.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:00:40.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:00:40.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:00:40.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:00:40.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:00:40.974 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:00:41.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:00:41.137 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:00:41.137 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:00:41.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:00:41.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:00:41.138 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:00:41.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855362 2026-02-13T02:00:41.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855362 2026-02-13T02:00:41.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-150323855362' 2026-02-13T02:00:41.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:00:41.212 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:00:41.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705676 2026-02-13T02:00:41.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705676 2026-02-13T02:00:41.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-150323855362 1-38654705676' 2026-02-13T02:00:41.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:00:41.285 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:00:41.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542156 2026-02-13T02:00:41.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542156 2026-02-13T02:00:41.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-150323855362 1-38654705676 2-60129542156' 2026-02-13T02:00:41.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:00:41.367 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-150323855362 2026-02-13T02:00:41.367 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:00:41.368 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:00:41.368 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-150323855362 2026-02-13T02:00:41.369 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:00:41.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855362 2026-02-13T02:00:41.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2026-02-13T02:00:41.370 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 150323855362 2026-02-13T02:00:41.371 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:00:41.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 150323855362 2026-02-13T02:00:41.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:00:42.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:00:42.540 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:00:42.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 150323855362 2026-02-13T02:00:42.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:00:43.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:00:43.703 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:00:43.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855362 -lt 150323855362 2026-02-13T02:00:43.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:00:43.874 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705676 2026-02-13T02:00:43.874 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:00:43.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:00:43.876 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705676 2026-02-13T02:00:43.876 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:00:43.878 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705676 2026-02-13T02:00:43.878 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705676' 2026-02-13T02:00:43.878 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 38654705676 2026-02-13T02:00:43.878 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:00:44.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705677 -lt 38654705676 2026-02-13T02:00:44.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:00:44.039 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542156 2026-02-13T02:00:44.039 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:00:44.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:00:44.041 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542156 2026-02-13T02:00:44.041 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:00:44.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542156 2026-02-13T02:00:44.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542156' 2026-02-13T02:00:44.043 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 60129542156 2026-02-13T02:00:44.044 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:00:44.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542156 -lt 60129542156 2026-02-13T02:00:44.207 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:00:44.208 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:00:44.208 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:00:44.402 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:00:44.402 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:00:44.403 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:00:44.403 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:00:44.403 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:00:44.403 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:00:44.404 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:00:44.404 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:00:44.558 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:00:44.558 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:00:44.559 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:00:44.559 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:00:44.748 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:00:44.748 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:00:44.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:00:44.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:492: TEST_auto_repair_bluestore_tag: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-13T02:00:44.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_tag: grep scrub_finish td/osd-scrub-repair/osd.1.log 2026-02-13T02:00:44.750 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T02:00:31.889+0000 7f7510665640 10 osd.1 pg_epoch: 26 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=15/15 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep(periodic-regular,auto-repair)] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2026-02-13T02:00:44.751 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T02:00:31.889+0000 7f7510665640 10 osd.1 pg_epoch: 26 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=15/15 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep(periodic-regular,auto-repair)] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2026-02-13T02:00:44.751 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T02:00:31.889+0000 7f7510665640 15 osd.1 pg_epoch: 26 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=15/15 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2026-02-13T02:00:44.751 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T02:00:31.889+0000 7f7510665640 20 osd.1 pg_epoch: 26 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=15/15 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2026-02-13T02:00:44.751 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T02:00:31.889+0000 7f7510665640 19 osd.1 pg_epoch: 26 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=15/15 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2026-02-13T02:00:44.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-13T02:00:44.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:00:44.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:00:44.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:00:44.752 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:00:44.752 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:00:44.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:00:44.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:00:44.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:00:44.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:00:44.862 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:00:44.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:00:44.864 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:00:44.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:00:44.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:00:44.866 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:00:44.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:00:44.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:00:44.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:00:44.868 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:00:44.869 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:00:44.870 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:00:44.870 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:00:44.880 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:00:44.880 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:00:44.880 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:00:44.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:00:44.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:00:44.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:00:44.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-13T02:00:44.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-13T02:00:44.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-13T02:00:44.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-13T02:00:44.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:00:44.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:00:44.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:00:44.883 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:00:44.884 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:00:44.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:00:44.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:00:44.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:00:44.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:00:44.886 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:00:44.886 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:00:44.887 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:00:44.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:00:44.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:00:44.888 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:00:44.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:00:44.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:00:44.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:00:44.890 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:00:44.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:00:44.892 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:00:44.893 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:00:44.894 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:00:44.894 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:00:44.894 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:00:44.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:00:44.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:00:44.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:00:44.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-13T02:00:44.897 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T02:00:44.897 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:00:44.897 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:00:44.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.35207 2026-02-13T02:00:44.899 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T02:00:44.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-13T02:00:44.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T02:00:44.901 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-13T02:00:44.901 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_erasure_coded_appends td/osd-scrub-repair 2026-02-13T02:00:44.901 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:369: TEST_auto_repair_erasure_coded_appends: auto_repair_erasure_coded td/osd-scrub-repair false 2026-02-13T02:00:44.901 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:328: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2026-02-13T02:00:44.901 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:329: auto_repair_erasure_coded: local allow_overwrites=false 2026-02-13T02:00:44.901 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:330: auto_repair_erasure_coded: local poolname=ecpool 2026-02-13T02:00:44.901 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2026-02-13T02:00:44.901 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-13T02:00:44.901 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-13T02:00:44.901 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-13T02:00:44.901 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-13T02:00:44.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-13T02:00:44.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-13T02:00:44.922 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-13T02:00:44.922 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:00:44.922 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:00:44.922 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:00:44.922 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:00:44.923 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:00:44.923 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:00:44.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-13T02:00:44.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-13T02:00:44.954 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-13T02:00:44.954 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:00:44.954 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:00:44.954 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-13T02:00:44.955 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-13T02:00:44.955 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:00:44.955 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:00:44.956 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:00:44.956 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:00:44.956 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:00:44.956 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:00:44.956 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:00:44.956 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:00:44.957 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get fsid 2026-02-13T02:00:45.013 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-13T02:00:45.013 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:00:45.013 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:00:45.013 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-13T02:00:45.013 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-13T02:00:45.015 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:00:45.015 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:00:45.015 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:00:45.015 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:00:45.015 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:00:45.015 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:00:45.015 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:00:45.015 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:00:45.015 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get mon_host 2026-02-13T02:00:45.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:334: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2026-02-13T02:00:45.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-13T02:00:45.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-13T02:00:45.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-13T02:00:45.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-13T02:00:45.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-13T02:00:45.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-13T02:00:45.166 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-13T02:00:45.166 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:00:45.166 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:00:45.167 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:00:45.167 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:00:45.167 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:00:45.167 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:00:45.167 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:00:45.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:00:45.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2026-02-13T02:00:45.187 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: auto_repair_erasure_coded: seq 0 2 2026-02-13T02:00:45.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: auto_repair_erasure_coded: for id in $(seq 0 2) 2026-02-13T02:00:45.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-13T02:00:45.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:00:45.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:00:45.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-13T02:00:45.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:00:45.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:00:45.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:00:45.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:00:45.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:00:45.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:00:45.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:00:45.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:00:45.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:00:45.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:00:45.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:00:45.193 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:00:45.193 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:00:45.193 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:00:45.193 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:00:45.193 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:00:45.193 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:00:45.193 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:00:45.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:00:45.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:00:45.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:00:45.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:00:45.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:00:45.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:00:45.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:00:45.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:00:45.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:00:45.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:00:45.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:00:45.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2026-02-13T02:00:45.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:00:45.195 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:00:45.195 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f8e11523-6337-4e18-a0c8-ab0afbd6f85c 2026-02-13T02:00:45.195 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 f8e11523-6337-4e18-a0c8-ab0afbd6f85c' 2026-02-13T02:00:45.196 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 f8e11523-6337-4e18-a0c8-ab0afbd6f85c 2026-02-13T02:00:45.196 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:00:45.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDNhY5pFqb6CxAAOQhwM3v0eeiqHHgnQErWWw== 2026-02-13T02:00:45.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDNhY5pFqb6CxAAOQhwM3v0eeiqHHgnQErWWw=="}' 2026-02-13T02:00:45.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f8e11523-6337-4e18-a0c8-ab0afbd6f85c -i td/osd-scrub-repair/0/new.json 2026-02-13T02:00:45.296 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:00:45.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-13T02:00:45.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDNhY5pFqb6CxAAOQhwM3v0eeiqHHgnQErWWw== --osd-uuid f8e11523-6337-4e18-a0c8-ab0afbd6f85c 2026-02-13T02:00:45.327 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:45.321+0000 7fdf0e753a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:45.328 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:45.325+0000 7fdf0e753a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:45.329 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:45.325+0000 7fdf0e753a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:45.330 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:45.325+0000 7fdf0e753a40 -1 bdev(0x55a65068cc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-13T02:00:45.330 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:45.325+0000 7fdf0e753a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-13T02:00:46.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-13T02:00:46.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:00:46.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-13T02:00:46.082 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T02:00:46.082 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:00:46.186 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T02:00:46.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-13T02:00:46.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-13T02:00:46.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:00:46.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:00:46.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:00:46.212 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:46.205+0000 7f466140ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:46.214 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:46.209+0000 7f466140ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:46.215 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:46.209+0000 7f466140ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:46.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-13T02:00:46.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:00:46.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:00:46.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:00:46.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:00:46.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:00:46.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:00:46.281 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:00:46.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:00:46.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:00:46.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:00:46.746 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:46.741+0000 7f466140ea40 -1 Falling back to public interface 2026-02-13T02:00:47.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:00:47.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:00:47.375 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:00:47.376 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:00:47.376 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:00:47.376 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:00:47.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:00:47.715 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:47.709+0000 7f466140ea40 -1 osd.0 0 log_to_monitors true 2026-02-13T02:00:48.469 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:00:48.469 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:00:48.469 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:00:48.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:00:48.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:00:48.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:00:48.631 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/4009434881,v1:127.0.0.1:6801/4009434881] [v2:127.0.0.1:6802/4009434881,v1:127.0.0.1:6803/4009434881] exists,up f8e11523-6337-4e18-a0c8-ab0afbd6f85c 2026-02-13T02:00:48.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:00:48.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:00:48.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:00:48.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: auto_repair_erasure_coded: for id in $(seq 0 2) 2026-02-13T02:00:48.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-13T02:00:48.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:00:48.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:00:48.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-13T02:00:48.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:00:48.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:00:48.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:00:48.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:00:48.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:00:48.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:00:48.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:00:48.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:00:48.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:00:48.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:00:48.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:00:48.632 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:00:48.632 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:00:48.632 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:00:48.632 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:00:48.632 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:00:48.632 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:00:48.632 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:00:48.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:00:48.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:00:48.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:00:48.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:00:48.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:00:48.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:00:48.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:00:48.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:00:48.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:00:48.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:00:48.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:00:48.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2026-02-13T02:00:48.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:00:48.635 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:00:48.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0e572043-7a03-423c-8193-002d1f997ba5 2026-02-13T02:00:48.636 INFO:tasks.workunit.client.0.trial112.stdout:add osd1 0e572043-7a03-423c-8193-002d1f997ba5 2026-02-13T02:00:48.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 0e572043-7a03-423c-8193-002d1f997ba5' 2026-02-13T02:00:48.636 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:00:48.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDQhY5paC2QJhAAdEWynkCpkdVkcHjVvd5gSw== 2026-02-13T02:00:48.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDQhY5paC2QJhAAdEWynkCpkdVkcHjVvd5gSw=="}' 2026-02-13T02:00:48.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0e572043-7a03-423c-8193-002d1f997ba5 -i td/osd-scrub-repair/1/new.json 2026-02-13T02:00:48.813 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:00:48.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-13T02:00:48.823 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDQhY5paC2QJhAAdEWynkCpkdVkcHjVvd5gSw== --osd-uuid 0e572043-7a03-423c-8193-002d1f997ba5 2026-02-13T02:00:48.840 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:48.833+0000 7f503d5a3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:48.842 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:48.837+0000 7f503d5a3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:48.843 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:48.837+0000 7f503d5a3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:48.843 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:48.837+0000 7f503d5a3a40 -1 bdev(0x561cb76e0c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-13T02:00:48.844 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:48.837+0000 7f503d5a3a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-13T02:00:49.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-13T02:00:49.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:00:49.481 INFO:tasks.workunit.client.0.trial112.stdout:adding osd1 key to auth repository 2026-02-13T02:00:49.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-13T02:00:49.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:00:49.683 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T02:00:49.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-13T02:00:49.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-13T02:00:49.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:00:49.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:00:49.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:00:49.709 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:49.705+0000 7f2b6bd41a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:49.711 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:49.705+0000 7f2b6bd41a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:49.712 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:49.705+0000 7f2b6bd41a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:49.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-13T02:00:49.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:00:49.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:00:49.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:00:49.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:00:49.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:00:49.843 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:00:49.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:00:49.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:00:49.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:00:50.006 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:00:50.114 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:50.109+0000 7f2b6bd41a40 -1 Falling back to public interface 2026-02-13T02:00:50.769 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:50.761+0000 7f2b6bd41a40 -1 osd.1 0 log_to_monitors true 2026-02-13T02:00:51.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:00:51.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:00:51.008 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:00:51.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:00:51.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:00:51.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:00:51.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:00:52.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:00:52.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:00:52.175 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:00:52.175 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:00:52.175 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:00:52.176 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:00:52.336 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2159009502,v1:127.0.0.1:6811/2159009502] [v2:127.0.0.1:6812/2159009502,v1:127.0.0.1:6813/2159009502] exists,up 0e572043-7a03-423c-8193-002d1f997ba5 2026-02-13T02:00:52.339 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:00:52.339 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:00:52.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:00:52.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: auto_repair_erasure_coded: for id in $(seq 0 2) 2026-02-13T02:00:52.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-13T02:00:52.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:00:52.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:00:52.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-13T02:00:52.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:00:52.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-13T02:00:52.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:00:52.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:00:52.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:00:52.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:00:52.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-13T02:00:52.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-13T02:00:52.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:00:52.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:00:52.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:00:52.341 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:00:52.341 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:00:52.341 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:00:52.341 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:00:52.341 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:00:52.341 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:00:52.341 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:00:52.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:00:52.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:00:52.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:00:52.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:00:52.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:00:52.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:00:52.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:00:52.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:00:52.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:00:52.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:00:52.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:00:52.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2026-02-13T02:00:52.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-13T02:00:52.343 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:00:52.343 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d5a7cfa4-4df3-4df7-9fc4-d87dc44ead01 2026-02-13T02:00:52.344 INFO:tasks.workunit.client.0.trial112.stdout:add osd2 d5a7cfa4-4df3-4df7-9fc4-d87dc44ead01 2026-02-13T02:00:52.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 d5a7cfa4-4df3-4df7-9fc4-d87dc44ead01' 2026-02-13T02:00:52.344 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:00:52.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDUhY5pH7nkFBAAs3op/SJlyyAgY5l+ZDag6Q== 2026-02-13T02:00:52.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDUhY5pH7nkFBAAs3op/SJlyyAgY5l+ZDag6Q=="}' 2026-02-13T02:00:52.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d5a7cfa4-4df3-4df7-9fc4-d87dc44ead01 -i td/osd-scrub-repair/2/new.json 2026-02-13T02:00:52.506 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:00:52.516 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-13T02:00:52.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDUhY5pH7nkFBAAs3op/SJlyyAgY5l+ZDag6Q== --osd-uuid d5a7cfa4-4df3-4df7-9fc4-d87dc44ead01 2026-02-13T02:00:52.535 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:52.529+0000 7f2d24c41a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:52.536 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:52.529+0000 7f2d24c41a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:52.537 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:52.533+0000 7f2d24c41a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:52.538 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:52.533+0000 7f2d24c41a40 -1 bdev(0x563eb86b4c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-13T02:00:52.538 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:52.533+0000 7f2d24c41a40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-13T02:00:53.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-13T02:00:53.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:00:53.180 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-13T02:00:53.181 INFO:tasks.workunit.client.0.trial112.stdout:adding osd2 key to auth repository 2026-02-13T02:00:53.181 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:00:53.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-13T02:00:53.378 INFO:tasks.workunit.client.0.trial112.stdout:start osd.2 2026-02-13T02:00:53.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-13T02:00:53.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:00:53.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:00:53.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:00:53.404 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:53.397+0000 7f499bf64a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:53.406 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:53.401+0000 7f499bf64a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:53.407 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:53.401+0000 7f499bf64a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:00:53.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-13T02:00:53.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:00:53.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T02:00:53.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:00:53.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:00:53.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:00:53.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:00:53.542 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:00:53.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:00:53.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:00:53.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:00:53.822 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:53.817+0000 7f499bf64a40 -1 Falling back to public interface 2026-02-13T02:00:54.430 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:00:54.425+0000 7f499bf64a40 -1 osd.2 0 log_to_monitors true 2026-02-13T02:00:54.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:00:54.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:00:54.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:00:54.700 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:00:54.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:00:54.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:00:54.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:00:55.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:00:55.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:00:55.865 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:00:55.866 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:00:55.866 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:00:55.866 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:00:56.020 INFO:tasks.workunit.client.0.trial112.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2772196559,v1:127.0.0.1:6819/2772196559] [v2:127.0.0.1:6820/2772196559,v1:127.0.0.1:6821/2772196559] exists,up d5a7cfa4-4df3-4df7-9fc4-d87dc44ead01 2026-02-13T02:00:56.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:00:56.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:00:56.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:00:56.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: auto_repair_erasure_coded: create_rbd_pool 2026-02-13T02:00:56.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-13T02:00:56.168 INFO:tasks.workunit.client.0.trial112.stderr:pool 'rbd' does not exist 2026-02-13T02:00:56.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-13T02:00:56.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-13T02:00:56.400 INFO:tasks.workunit.client.0.trial112.stderr:pool 'rbd' created 2026-02-13T02:00:56.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:00:57.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-13T02:00:57.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:344: auto_repair_erasure_coded: wait_for_clean 2026-02-13T02:00:57.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:00:57.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:00:57.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:00:57.690 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:00:57.690 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:00:57.690 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:00:57.691 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:00:57.691 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:00:57.691 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:00:57.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:00:57.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:00:57.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:00:57.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:00:57.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:00:57.804 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:00:57.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:00:57.965 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:00:57.965 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:00:57.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:00:57.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:00:57.966 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:00:58.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-13T02:00:58.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-13T02:00:58.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-13T02:00:58.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:00:58.041 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:00:58.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672963 2026-02-13T02:00:58.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672963 2026-02-13T02:00:58.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-42949672963' 2026-02-13T02:00:58.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:00:58.113 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:00:58.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542146 2026-02-13T02:00:58.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542146 2026-02-13T02:00:58.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-42949672963 2-60129542146' 2026-02-13T02:00:58.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:00:58.189 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-13T02:00:58.189 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:00:58.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:00:58.190 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-13T02:00:58.191 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:00:58.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-13T02:00:58.192 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836483 2026-02-13T02:00:58.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-13T02:00:58.193 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:00:58.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-13T02:00:58.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:00:59.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:00:59.365 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:00:59.527 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836483 2026-02-13T02:00:59.527 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:00:59.528 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672963 2026-02-13T02:00:59.528 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:00:59.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:00:59.530 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672963 2026-02-13T02:00:59.530 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:00:59.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672963 2026-02-13T02:00:59.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2026-02-13T02:00:59.532 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 42949672963 2026-02-13T02:00:59.532 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:00:59.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672963 -lt 42949672963 2026-02-13T02:00:59.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:00:59.702 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542146 2026-02-13T02:00:59.703 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:00:59.704 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:00:59.705 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542146 2026-02-13T02:00:59.705 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:00:59.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542146 2026-02-13T02:00:59.706 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542146 2026-02-13T02:00:59.707 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542146' 2026-02-13T02:00:59.707 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:00:59.866 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542146 -lt 60129542146 2026-02-13T02:00:59.867 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:00:59.867 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:00:59.868 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:01:00.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-13T02:01:00.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:01:00.062 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:01:00.062 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:01:00.062 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:01:00.062 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:01:00.062 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:01:00.062 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:01:00.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-13T02:01:00.228 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:01:00.228 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:01:00.228 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:01:00.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-13T02:01:00.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:01:00.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:01:00.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:347: auto_repair_erasure_coded: create_ec_pool ecpool false k=2 m=1 2026-02-13T02:01:00.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2522: create_ec_pool: local pool_name=ecpool 2026-02-13T02:01:00.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2523: create_ec_pool: shift 2026-02-13T02:01:00.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2524: create_ec_pool: local allow_overwrites=false 2026-02-13T02:01:00.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2525: create_ec_pool: shift 2026-02-13T02:01:00.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2527: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2026-02-13T02:01:00.645 INFO:tasks.workunit.client.0.trial112.stderr:load: isa 2026-02-13T02:01:00.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2529: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2026-02-13T02:01:00.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2026-02-13T02:01:00.950 INFO:tasks.workunit.client.0.trial112.stderr:pool 'ecpool' created 2026-02-13T02:01:00.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:01:01.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2531: create_ec_pool: '[' false = true ']' 2026-02-13T02:01:01.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2535: create_ec_pool: wait_for_clean 2026-02-13T02:01:01.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:01:01.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:01:01.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:01:01.965 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:01:01.965 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:01:01.966 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:01:01.966 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:01:01.966 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:01:01.967 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:01:02.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:01:02.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:01:02.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:01:02.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:01:02.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:01:02.071 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:01:02.232 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:01:02.233 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:01:02.233 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:01:02.233 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:01:02.233 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:01:02.233 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:01:02.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836485 2026-02-13T02:01:02.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836485 2026-02-13T02:01:02.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485' 2026-02-13T02:01:02.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:01:02.305 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:01:02.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672965 2026-02-13T02:01:02.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672965 2026-02-13T02:01:02.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-42949672965' 2026-02-13T02:01:02.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:01:02.379 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:01:02.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542148 2026-02-13T02:01:02.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542148 2026-02-13T02:01:02.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-42949672965 2-60129542148' 2026-02-13T02:01:02.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:01:02.453 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836485 2026-02-13T02:01:02.453 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:01:02.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:01:02.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836485 2026-02-13T02:01:02.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:01:02.456 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836485 2026-02-13T02:01:02.456 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-13T02:01:02.457 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836485 2026-02-13T02:01:02.457 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:01:02.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836485 2026-02-13T02:01:02.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:01:03.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:01:03.624 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:01:03.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 21474836485 2026-02-13T02:01:03.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:01:03.789 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672965 2026-02-13T02:01:03.789 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:01:03.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:01:03.791 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672965 2026-02-13T02:01:03.792 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:01:03.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672965 2026-02-13T02:01:03.793 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 42949672965 2026-02-13T02:01:03.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2026-02-13T02:01:03.794 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:01:03.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672965 -lt 42949672965 2026-02-13T02:01:03.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:01:03.956 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542148 2026-02-13T02:01:03.956 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:01:03.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:01:03.958 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542148 2026-02-13T02:01:03.958 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:01:03.959 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542148 2026-02-13T02:01:03.960 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542148 2026-02-13T02:01:03.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542148' 2026-02-13T02:01:03.960 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:01:04.122 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542148 -lt 60129542148 2026-02-13T02:01:04.123 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:01:04.123 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:01:04.124 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:01:04.319 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:01:04.319 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:01:04.320 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:01:04.320 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:01:04.320 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:01:04.320 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:01:04.321 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:01:04.321 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:01:04.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:01:04.481 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:01:04.481 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:01:04.482 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:01:04.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:01:04.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:01:04.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:01:04.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2536: create_ec_pool: return 0 2026-02-13T02:01:04.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:350: auto_repair_erasure_coded: local payload=ABCDEF 2026-02-13T02:01:04.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:351: auto_repair_erasure_coded: echo ABCDEF 2026-02-13T02:01:04.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-13T02:01:04.703 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:356: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2026-02-13T02:01:04.703 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=ecpool 2026-02-13T02:01:04.703 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-13T02:01:04.704 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary ecpool SOMETHING 2026-02-13T02:01:04.704 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2026-02-13T02:01:04.704 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-13T02:01:04.704 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2026-02-13T02:01:04.704 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T02:01:04.862 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=2 2026-02-13T02:01:04.862 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map ecpool SOMETHING 2026-02-13T02:01:04.862 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2026-02-13T02:01:05.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:356: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2026-02-13T02:01:05.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:01:05.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:01:05.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:01:05.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:01:05.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2026-02-13T02:01:05.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:01:05.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:01:05.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:01:05.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:01:05.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:01:05.028 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:01:05.028 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:01:05.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:01:05.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:01:05.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:01:05.336 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:01:05.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2026-02-13T02:01:05.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:01:05.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:01:05.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:01:05.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:01:05.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:01:05.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2026-02-13T02:01:05.521 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.0_head,#-3:00000000:::deep_scrub_1.0:head#, (61) No data available 2026-02-13T02:01:05.521 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2026-02-13T02:01:05.522 INFO:tasks.workunit.client.0.trial112.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2026-02-13T02:01:05.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-13T02:01:05.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:01:05.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:01:05.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:01:05.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:01:05.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:01:05.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:01:05.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:01:05.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:01:05.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:01:05.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:01:05.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:01:05.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:01:05.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:01:05.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:01:05.638 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:01:05.638 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:01:05.638 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:01:05.638 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:01:05.638 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:01:05.638 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:01:05.638 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:01:05.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:01:05.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:01:05.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:01:05.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:01:05.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:01:05.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:01:05.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:01:05.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:01:05.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:01:05.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2026-02-13T02:01:05.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:01:05.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:01:05.640 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:01:05.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-13T02:01:05.640 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:01:05.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:01:05.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:01:05.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:01:05.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:01:05.660 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:05.652+0000 7f1243473a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:01:05.662 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:05.656+0000 7f1243473a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:01:05.663 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:05.656+0000 7f1243473a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:01:05.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:01:05.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:01:05.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:01:05.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:01:05.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:01:05.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:01:05.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:01:05.804 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:01:05.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:01:05.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:01:05.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:01:06.015 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:06.008+0000 7f1243473a40 -1 Falling back to public interface 2026-02-13T02:01:06.600 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:06.592+0000 7f1243473a40 -1 osd.1 27 log_to_monitors true 2026-02-13T02:01:06.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:01:06.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:01:06.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:01:06.963 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:01:06.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:01:06.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:01:07.130 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 31 up_thru 31 down_at 28 last_clean_interval [10,27) [v2:127.0.0.1:6810/1837819668,v1:127.0.0.1:6811/1837819668] [v2:127.0.0.1:6812/1837819668,v1:127.0.0.1:6813/1837819668] exists,up 0e572043-7a03-423c-8193-002d1f997ba5 2026-02-13T02:01:07.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:01:07.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:01:07.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:01:07.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:01:07.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:01:07.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:01:07.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:01:07.131 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:01:07.131 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:01:07.132 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:01:07.132 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:01:07.132 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:01:07.132 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:01:07.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:01:07.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:01:07.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:01:07.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:01:07.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:01:07.241 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:01:07.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:01:07.408 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:01:07.409 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:01:07.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:01:07.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:01:07.409 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:01:07.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836487 2026-02-13T02:01:07.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836487 2026-02-13T02:01:07.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487' 2026-02-13T02:01:07.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:01:07.483 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:01:07.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986178 2026-02-13T02:01:07.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986178 2026-02-13T02:01:07.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487 1-133143986178' 2026-02-13T02:01:07.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:01:07.554 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:01:07.625 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542150 2026-02-13T02:01:07.625 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542150 2026-02-13T02:01:07.625 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487 1-133143986178 2-60129542150' 2026-02-13T02:01:07.625 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:01:07.626 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836487 2026-02-13T02:01:07.626 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:01:07.627 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:01:07.628 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836487 2026-02-13T02:01:07.628 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:01:07.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836487 2026-02-13T02:01:07.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2026-02-13T02:01:07.629 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 21474836487 2026-02-13T02:01:07.630 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:01:07.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836487 2026-02-13T02:01:07.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:01:08.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:01:08.795 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:01:08.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836487 2026-02-13T02:01:08.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:01:09.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:01:09.955 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:01:10.121 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836487 2026-02-13T02:01:10.121 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:01:10.122 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-133143986178 2026-02-13T02:01:10.122 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:01:10.123 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:01:10.124 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-133143986178 2026-02-13T02:01:10.124 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:01:10.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=133143986178 2026-02-13T02:01:10.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 133143986178' 2026-02-13T02:01:10.126 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 133143986178 2026-02-13T02:01:10.126 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:01:10.291 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 133143986178 -lt 133143986178 2026-02-13T02:01:10.291 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:01:10.292 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542150 2026-02-13T02:01:10.292 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:01:10.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:01:10.294 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542150 2026-02-13T02:01:10.294 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:01:10.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542150 2026-02-13T02:01:10.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542150' 2026-02-13T02:01:10.296 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 60129542150 2026-02-13T02:01:10.297 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:01:10.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542150 -lt 60129542150 2026-02-13T02:01:10.473 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:01:10.473 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:01:10.474 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:01:10.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:01:10.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:01:10.674 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:01:10.674 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:01:10.675 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:01:10.675 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:01:10.675 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:01:10.675 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:01:10.836 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:01:10.837 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:01:10.837 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:01:10.837 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:01:11.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:01:11.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:01:11.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:01:11.032 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:358: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2026-02-13T02:01:11.032 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-13T02:01:11.032 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-13T02:01:11.033 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-13T02:01:11.033 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T02:01:11.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:358: auto_repair_erasure_coded: local pgid=2.0 2026-02-13T02:01:11.191 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:359: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2026-02-13T02:01:11.192 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:01:11.192 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:01:11.192 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:01:11.192 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:01:11.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:359: auto_repair_erasure_coded: wait_for_scrub 2.0 2026-02-13T02:01:00.946529+0000 2026-02-13T02:01:11.347 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-13T02:01:11.347 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:01:00.946529+0000 2026-02-13T02:01:11.347 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:01:11.347 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:01:11.347 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:01:11.347 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:01:11.347 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:01:11.347 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:01:11.347 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:01:11.347 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:01:11.502 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:01:00.946529+0000 '>' 2026-02-13T02:01:00.946529+0000 2026-02-13T02:01:11.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:01:12.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:01:12.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:01:12.504 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:01:12.505 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:01:12.505 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:01:12.505 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:01:12.505 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:01:12.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:01:00.946529+0000 '>' 2026-02-13T02:01:00.946529+0000 2026-02-13T02:01:12.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:01:13.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:01:13.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:01:13.665 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:01:13.665 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:01:13.665 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:01:13.665 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:01:13.665 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:01:13.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:01:00.946529+0000 '>' 2026-02-13T02:01:00.946529+0000 2026-02-13T02:01:13.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:01:14.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:01:14.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:01:14.820 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:01:14.820 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:01:14.821 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:01:14.821 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:01:14.821 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:01:14.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:01:00.946529+0000 '>' 2026-02-13T02:01:00.946529+0000 2026-02-13T02:01:14.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:01:15.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:01:15.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:01:15.986 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:01:15.986 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:01:15.986 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:01:15.986 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:01:15.986 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:01:16.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:01:00.946529+0000 '>' 2026-02-13T02:01:00.946529+0000 2026-02-13T02:01:16.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:01:17.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:01:17.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:01:17.151 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:01:17.151 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:01:17.151 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:01:17.151 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:01:17.151 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:01:17.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:01:00.946529+0000 '>' 2026-02-13T02:01:00.946529+0000 2026-02-13T02:01:17.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:01:18.315 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:01:18.315 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:01:18.316 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:01:18.316 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:01:18.316 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:01:18.317 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:01:18.317 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:01:18.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:01:13.426590+0000 '>' 2026-02-13T02:01:00.946529+0000 2026-02-13T02:01:18.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:01:18.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:360: auto_repair_erasure_coded: wait_for_clean 2026-02-13T02:01:18.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:01:18.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:01:18.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:01:18.479 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:01:18.479 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:01:18.479 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:01:18.480 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:01:18.480 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:01:18.480 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:01:18.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:01:18.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:01:18.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:01:18.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:01:18.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:01:18.579 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:01:18.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:01:18.744 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:01:18.744 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:01:18.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:01:18.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:01:18.745 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:01:18.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836490 2026-02-13T02:01:18.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836490 2026-02-13T02:01:18.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836490' 2026-02-13T02:01:18.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:01:18.820 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:01:18.892 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986181 2026-02-13T02:01:18.892 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986181 2026-02-13T02:01:18.892 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836490 1-133143986181' 2026-02-13T02:01:18.892 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:01:18.892 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:01:18.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542153 2026-02-13T02:01:18.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542153 2026-02-13T02:01:18.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836490 1-133143986181 2-60129542153' 2026-02-13T02:01:18.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:01:18.965 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836490 2026-02-13T02:01:18.965 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:01:18.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:01:18.967 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836490 2026-02-13T02:01:18.967 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:01:18.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836490 2026-02-13T02:01:18.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2026-02-13T02:01:18.969 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836490 2026-02-13T02:01:18.969 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:01:19.138 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836490 2026-02-13T02:01:19.138 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:01:20.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:01:20.140 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:01:20.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836490 2026-02-13T02:01:20.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:01:20.301 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-133143986181 2026-02-13T02:01:20.301 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:01:20.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:01:20.303 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-133143986181 2026-02-13T02:01:20.303 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:01:20.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=133143986181 2026-02-13T02:01:20.305 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 133143986181 2026-02-13T02:01:20.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 133143986181' 2026-02-13T02:01:20.305 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:01:20.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 133143986181 -lt 133143986181 2026-02-13T02:01:20.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:01:20.482 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542153 2026-02-13T02:01:20.482 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:01:20.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:01:20.484 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542153 2026-02-13T02:01:20.484 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:01:20.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542153 2026-02-13T02:01:20.486 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542153 2026-02-13T02:01:20.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542153' 2026-02-13T02:01:20.486 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:01:20.646 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542153 -lt 60129542153 2026-02-13T02:01:20.646 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:01:20.647 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:01:20.647 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:01:20.839 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:01:20.839 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:01:20.839 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:01:20.839 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:01:20.839 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:01:20.839 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:01:20.840 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:01:20.840 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:01:20.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:01:20.994 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:01:20.995 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:01:20.995 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:01:21.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:01:21.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:01:21.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:01:21.188 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2026-02-13T02:01:21.188 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=ecpool 2026-02-13T02:01:21.188 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-13T02:01:21.189 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary ecpool SOMETHING 2026-02-13T02:01:21.189 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2026-02-13T02:01:21.190 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-13T02:01:21.190 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2026-02-13T02:01:21.190 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T02:01:21.352 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=2 2026-02-13T02:01:21.352 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map ecpool SOMETHING 2026-02-13T02:01:21.352 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2026-02-13T02:01:21.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-13T02:01:21.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:01:21.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:01:21.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:01:21.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:01:21.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-13T02:01:21.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:01:21.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:01:21.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:01:21.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:01:21.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:01:21.519 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:01:21.519 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:01:21.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:01:21.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:01:21.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:01:21.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:01:21.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-13T02:01:21.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:01:21.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:01:21.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:01:21.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:01:21.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:01:21.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2026-02-13T02:01:21.884 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::deep_scrub_1.3:head#, (61) No data available 2026-02-13T02:01:21.884 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2026-02-13T02:01:21.885 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.0_head,#-3:00000000:::deep_scrub_1.0:head#, (61) No data available 2026-02-13T02:01:21.885 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2026-02-13T02:01:21.885 INFO:tasks.workunit.client.0.trial112.stdout:_ 2026-02-13T02:01:21.885 INFO:tasks.workunit.client.0.trial112.stdout:hinfo_key 2026-02-13T02:01:21.885 INFO:tasks.workunit.client.0.trial112.stdout:snapset 2026-02-13T02:01:21.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-13T02:01:21.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:01:21.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:01:21.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:01:21.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:01:21.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:01:21.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:01:21.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:01:21.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:01:21.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:01:21.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:01:21.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:01:21.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:01:21.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:01:21.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:01:21.996 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:01:21.996 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:01:21.996 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:01:21.996 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:01:21.996 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:01:21.996 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:01:21.996 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:01:21.998 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:01:21.998 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:01:21.998 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:01:21.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:01:21.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:01:21.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:01:21.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:01:21.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:01:21.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:01:21.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2026-02-13T02:01:21.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:01:21.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:01:21.999 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:01:21.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-13T02:01:22.000 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:01:22.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:01:22.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:01:22.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:01:22.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:01:22.015 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:22.008+0000 7f403a764a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:01:22.017 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:22.008+0000 7f403a764a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:01:22.018 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:22.012+0000 7f403a764a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:01:22.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:01:22.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:01:22.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:01:22.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:01:22.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:01:22.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:01:22.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:01:22.158 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:01:22.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:01:22.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:01:22.319 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:01:22.395 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:22.388+0000 7f403a764a40 -1 Falling back to public interface 2026-02-13T02:01:23.045 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:23.036+0000 7f403a764a40 -1 osd.1 33 log_to_monitors true 2026-02-13T02:01:23.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:01:23.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:01:23.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:01:23.322 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:01:23.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:01:23.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:01:23.481 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 37 up_thru 37 down_at 34 last_clean_interval [31,33) [v2:127.0.0.1:6810/3561948071,v1:127.0.0.1:6811/3561948071] [v2:127.0.0.1:6812/3561948071,v1:127.0.0.1:6813/3561948071] exists,up 0e572043-7a03-423c-8193-002d1f997ba5 2026-02-13T02:01:23.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:01:23.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:01:23.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:01:23.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:01:23.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:01:23.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:01:23.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:01:23.482 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:01:23.483 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:01:23.483 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:01:23.483 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:01:23.483 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:01:23.483 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:01:23.601 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:01:23.601 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:01:23.601 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:01:23.601 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:01:23.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:01:23.602 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:01:23.768 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:01:23.768 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:01:23.768 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:01:23.768 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:01:23.768 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:01:23.769 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:01:23.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836492 2026-02-13T02:01:23.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836492 2026-02-13T02:01:23.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836492' 2026-02-13T02:01:23.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:01:23.842 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:01:23.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789954 2026-02-13T02:01:23.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789954 2026-02-13T02:01:23.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836492 1-158913789954' 2026-02-13T02:01:23.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:01:23.923 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:01:23.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542155 2026-02-13T02:01:23.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542155 2026-02-13T02:01:24.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836492 1-158913789954 2-60129542155' 2026-02-13T02:01:24.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:01:24.000 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836492 2026-02-13T02:01:24.000 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:01:24.001 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:01:24.002 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836492 2026-02-13T02:01:24.002 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:01:24.004 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836492 2026-02-13T02:01:24.004 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2026-02-13T02:01:24.004 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 21474836492 2026-02-13T02:01:24.005 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:01:24.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836492 2026-02-13T02:01:24.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:01:25.175 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:01:25.175 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:01:25.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836492 2026-02-13T02:01:25.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:01:26.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:01:26.342 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:01:26.505 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836493 -lt 21474836492 2026-02-13T02:01:26.505 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:01:26.506 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-158913789954 2026-02-13T02:01:26.506 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:01:26.507 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:01:26.507 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-158913789954 2026-02-13T02:01:26.508 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:01:26.509 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789954 2026-02-13T02:01:26.509 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 158913789954' 2026-02-13T02:01:26.509 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 158913789954 2026-02-13T02:01:26.510 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:01:26.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789954 -lt 158913789954 2026-02-13T02:01:26.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:01:26.684 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542155 2026-02-13T02:01:26.684 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:01:26.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:01:26.686 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542155 2026-02-13T02:01:26.686 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:01:26.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542155 2026-02-13T02:01:26.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542155' 2026-02-13T02:01:26.687 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 60129542155 2026-02-13T02:01:26.688 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:01:26.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542155 -lt 60129542155 2026-02-13T02:01:26.854 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:01:26.854 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:01:26.854 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:01:27.058 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:01:27.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:01:27.060 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:01:27.060 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:01:27.060 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:01:27.060 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:01:27.060 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:01:27.060 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:01:27.219 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:01:27.220 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:01:27.220 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:01:27.220 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:01:27.416 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:01:27.416 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:01:27.416 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:01:27.416 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-13T02:01:27.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-13T02:01:27.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-13T02:01:27.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:01:27.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:01:27.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:01:27.436 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:01:27.436 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:01:27.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:01:27.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:01:27.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:01:27.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:01:27.546 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:01:27.548 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:01:27.548 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:01:27.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:01:27.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:01:27.550 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:01:27.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:01:27.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:01:27.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:01:27.552 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:01:27.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:01:27.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:01:27.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:01:27.568 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:01:27.569 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:01:27.569 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:01:27.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:01:27.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:01:27.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:01:27.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-13T02:01:27.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-13T02:01:27.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-13T02:01:27.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-13T02:01:27.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:01:27.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:01:27.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:01:27.571 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:01:27.571 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:01:27.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:01:27.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:01:27.572 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:01:27.574 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:01:27.574 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:01:27.575 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:01:27.576 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:01:27.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:01:27.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:01:27.577 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:01:27.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:01:27.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:01:27.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:01:27.579 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:01:27.580 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:01:27.581 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:01:27.581 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:01:27.583 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:01:27.583 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:01:27.583 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:01:27.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:01:27.584 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:01:27.584 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:01:27.584 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-13T02:01:27.586 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T02:01:27.586 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:01:27.586 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:01:27.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.35207 2026-02-13T02:01:27.588 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T02:01:27.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-13T02:01:27.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T02:01:27.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-13T02:01:27.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_erasure_coded_overwrites td/osd-scrub-repair 2026-02-13T02:01:27.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:373: TEST_auto_repair_erasure_coded_overwrites: '[' true = true ']' 2026-02-13T02:01:27.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:374: TEST_auto_repair_erasure_coded_overwrites: auto_repair_erasure_coded td/osd-scrub-repair true 2026-02-13T02:01:27.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:328: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2026-02-13T02:01:27.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:329: auto_repair_erasure_coded: local allow_overwrites=true 2026-02-13T02:01:27.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:330: auto_repair_erasure_coded: local poolname=ecpool 2026-02-13T02:01:27.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2026-02-13T02:01:27.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-13T02:01:27.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-13T02:01:27.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-13T02:01:27.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-13T02:01:27.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-13T02:01:27.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-13T02:01:27.610 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-13T02:01:27.611 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:01:27.611 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:01:27.611 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:01:27.611 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:01:27.611 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:01:27.612 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:01:27.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-13T02:01:27.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-13T02:01:27.641 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-13T02:01:27.642 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:01:27.642 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:01:27.642 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-13T02:01:27.643 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-13T02:01:27.643 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:01:27.643 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:01:27.643 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:01:27.643 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:01:27.643 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:01:27.643 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:01:27.644 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:01:27.645 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:01:27.645 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get fsid 2026-02-13T02:01:27.699 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-13T02:01:27.700 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:01:27.700 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:01:27.700 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-13T02:01:27.700 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-13T02:01:27.701 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:01:27.701 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:01:27.701 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:01:27.702 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:01:27.702 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:01:27.702 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:01:27.702 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:01:27.702 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:01:27.702 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get mon_host 2026-02-13T02:01:27.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:334: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2026-02-13T02:01:27.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-13T02:01:27.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-13T02:01:27.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-13T02:01:27.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-13T02:01:27.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-13T02:01:27.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-13T02:01:27.853 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-13T02:01:27.853 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:01:27.853 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:01:27.853 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:01:27.853 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:01:27.853 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:01:27.854 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:01:27.854 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:01:27.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:01:27.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2026-02-13T02:01:27.875 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: auto_repair_erasure_coded: seq 0 2 2026-02-13T02:01:27.878 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: auto_repair_erasure_coded: for id in $(seq 0 2) 2026-02-13T02:01:27.878 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-13T02:01:27.878 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:01:27.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:01:27.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-13T02:01:27.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:01:27.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:01:27.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:01:27.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:01:27.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:01:27.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:01:27.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:01:27.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:01:27.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:01:27.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:01:27.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:01:27.880 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:01:27.880 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:01:27.880 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:01:27.880 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:01:27.880 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:01:27.880 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:01:27.880 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:01:27.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:01:27.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:01:27.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:01:27.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:01:27.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:01:27.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:01:27.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:01:27.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:01:27.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:01:27.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:01:27.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:01:27.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2026-02-13T02:01:27.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:01:27.882 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:01:27.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2cbf9a4b-ee2e-42e9-935e-90aebdfd75b7 2026-02-13T02:01:27.882 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 2cbf9a4b-ee2e-42e9-935e-90aebdfd75b7 2026-02-13T02:01:27.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 2cbf9a4b-ee2e-42e9-935e-90aebdfd75b7' 2026-02-13T02:01:27.883 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:01:27.892 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD3hY5pHnf2NBAA6oK7BaJr7GlLPa0taKMXhg== 2026-02-13T02:01:27.892 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD3hY5pHnf2NBAA6oK7BaJr7GlLPa0taKMXhg=="}' 2026-02-13T02:01:27.892 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2cbf9a4b-ee2e-42e9-935e-90aebdfd75b7 -i td/osd-scrub-repair/0/new.json 2026-02-13T02:01:27.984 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:01:27.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-13T02:01:27.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQD3hY5pHnf2NBAA6oK7BaJr7GlLPa0taKMXhg== --osd-uuid 2cbf9a4b-ee2e-42e9-935e-90aebdfd75b7 2026-02-13T02:01:28.014 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:28.008+0000 7fabd69b1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:01:28.015 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:28.008+0000 7fabd69b1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:01:28.016 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:28.008+0000 7fabd69b1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:01:28.017 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:28.008+0000 7fabd69b1a40 -1 bdev(0x5633bb2c0c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-13T02:01:28.017 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:28.008+0000 7fabd69b1a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-13T02:01:28.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-13T02:01:28.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:01:28.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-13T02:01:28.671 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T02:01:28.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:01:28.770 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T02:01:28.771 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-13T02:01:28.771 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-13T02:01:28.771 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:01:28.771 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:01:28.771 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:01:28.787 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:28.780+0000 7feae5f23a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:01:28.789 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:28.780+0000 7feae5f23a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:01:28.790 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:28.784+0000 7feae5f23a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:01:28.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-13T02:01:28.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:01:28.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:01:28.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:01:28.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:01:28.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:01:28.865 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:01:28.866 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:01:28.866 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:01:28.866 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:01:28.959 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:01:29.167 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:29.160+0000 7feae5f23a40 -1 Falling back to public interface 2026-02-13T02:01:29.803 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:29.795+0000 7feae5f23a40 -1 osd.0 0 log_to_monitors true 2026-02-13T02:01:29.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:01:29.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:01:29.961 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:01:29.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:01:29.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:01:29.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:01:30.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:01:31.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:01:31.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:01:31.078 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:01:31.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:01:31.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:01:31.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:01:31.235 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/418821983,v1:127.0.0.1:6801/418821983] [v2:127.0.0.1:6802/418821983,v1:127.0.0.1:6803/418821983] exists,up 2cbf9a4b-ee2e-42e9-935e-90aebdfd75b7 2026-02-13T02:01:31.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:01:31.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:01:31.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:01:31.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: auto_repair_erasure_coded: for id in $(seq 0 2) 2026-02-13T02:01:31.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-13T02:01:31.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:01:31.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:01:31.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-13T02:01:31.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:01:31.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:01:31.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:01:31.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:01:31.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:01:31.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:01:31.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:01:31.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:01:31.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:01:31.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:01:31.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:01:31.240 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:01:31.241 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:01:31.241 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:01:31.241 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:01:31.241 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:01:31.241 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:01:31.241 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:01:31.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:01:31.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:01:31.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:01:31.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:01:31.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:01:31.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:01:31.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:01:31.242 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:01:31.242 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:01:31.242 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:01:31.242 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:01:31.242 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2026-02-13T02:01:31.242 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:01:31.242 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:01:31.243 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=abae2a6d-0142-4540-a20c-ec3bdc796aaf 2026-02-13T02:01:31.243 INFO:tasks.workunit.client.0.trial112.stdout:add osd1 abae2a6d-0142-4540-a20c-ec3bdc796aaf 2026-02-13T02:01:31.243 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 abae2a6d-0142-4540-a20c-ec3bdc796aaf' 2026-02-13T02:01:31.243 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:01:31.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD7hY5pLwvNDhAAYAZ9qoe2DHcOMtODlrr7sA== 2026-02-13T02:01:31.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD7hY5pLwvNDhAAYAZ9qoe2DHcOMtODlrr7sA=="}' 2026-02-13T02:01:31.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new abae2a6d-0142-4540-a20c-ec3bdc796aaf -i td/osd-scrub-repair/1/new.json 2026-02-13T02:01:31.402 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:01:31.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-13T02:01:31.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQD7hY5pLwvNDhAAYAZ9qoe2DHcOMtODlrr7sA== --osd-uuid abae2a6d-0142-4540-a20c-ec3bdc796aaf 2026-02-13T02:01:31.430 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:31.419+0000 7f4d0bd60a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:01:31.431 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:31.423+0000 7f4d0bd60a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:01:31.432 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:31.423+0000 7f4d0bd60a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:01:31.433 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:31.423+0000 7f4d0bd60a40 -1 bdev(0x555d6772cc00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-13T02:01:31.433 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:31.423+0000 7f4d0bd60a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-13T02:01:32.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-13T02:01:32.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:01:32.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-13T02:01:32.078 INFO:tasks.workunit.client.0.trial112.stdout:adding osd1 key to auth repository 2026-02-13T02:01:32.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:01:32.281 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T02:01:32.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-13T02:01:32.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-13T02:01:32.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:01:32.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:01:32.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:01:32.308 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:32.299+0000 7fe5fd8afa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:01:32.309 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:32.299+0000 7fe5fd8afa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:01:32.310 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:32.303+0000 7fe5fd8afa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:01:32.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-13T02:01:32.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:01:32.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:01:32.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:01:32.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:01:32.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:01:32.444 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:01:32.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:01:32.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:01:32.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:01:32.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:01:32.699 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:32.691+0000 7fe5fd8afa40 -1 Falling back to public interface 2026-02-13T02:01:33.285 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:33.275+0000 7fe5fd8afa40 -1 osd.1 0 log_to_monitors true 2026-02-13T02:01:33.611 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:01:33.611 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:01:33.612 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:01:33.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:01:33.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:01:33.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:01:33.775 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:01:34.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:01:34.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:01:34.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:01:34.778 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:01:34.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:01:34.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:01:34.934 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4045243837,v1:127.0.0.1:6811/4045243837] [v2:127.0.0.1:6812/4045243837,v1:127.0.0.1:6813/4045243837] exists,up abae2a6d-0142-4540-a20c-ec3bdc796aaf 2026-02-13T02:01:34.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:01:34.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:01:34.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:01:34.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: auto_repair_erasure_coded: for id in $(seq 0 2) 2026-02-13T02:01:34.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-13T02:01:34.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:01:34.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:01:34.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-13T02:01:34.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:01:34.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-13T02:01:34.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:01:34.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:01:34.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:01:34.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:01:34.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-13T02:01:34.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-13T02:01:34.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:01:34.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:01:34.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:01:34.935 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:01:34.935 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:01:34.935 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:01:34.935 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:01:34.935 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:01:34.936 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:01:34.936 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:01:34.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:01:34.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:01:34.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:01:34.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:01:34.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:01:34.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:01:34.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:01:34.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:01:34.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:01:34.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:01:34.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:01:34.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2026-02-13T02:01:34.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-13T02:01:34.938 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:01:34.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=04e5adac-355e-48c7-a14a-582e1b533035 2026-02-13T02:01:34.938 INFO:tasks.workunit.client.0.trial112.stdout:add osd2 04e5adac-355e-48c7-a14a-582e1b533035 2026-02-13T02:01:34.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 04e5adac-355e-48c7-a14a-582e1b533035' 2026-02-13T02:01:34.939 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:01:34.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD+hY5p+ZF/OBAAEnRnEOrWqggXPFNdKpiwmA== 2026-02-13T02:01:34.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD+hY5p+ZF/OBAAEnRnEOrWqggXPFNdKpiwmA=="}' 2026-02-13T02:01:34.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 04e5adac-355e-48c7-a14a-582e1b533035 -i td/osd-scrub-repair/2/new.json 2026-02-13T02:01:35.119 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:01:35.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-13T02:01:35.132 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQD+hY5p+ZF/OBAAEnRnEOrWqggXPFNdKpiwmA== --osd-uuid 04e5adac-355e-48c7-a14a-582e1b533035 2026-02-13T02:01:35.157 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:35.147+0000 7fcf3ee5ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:01:35.159 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:35.151+0000 7fcf3ee5ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:01:35.160 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:35.151+0000 7fcf3ee5ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:01:35.160 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:35.151+0000 7fcf3ee5ba40 -1 bdev(0x5625d407ec00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-13T02:01:35.161 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:35.151+0000 7fcf3ee5ba40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-13T02:01:35.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-13T02:01:35.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:01:35.801 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-13T02:01:35.801 INFO:tasks.workunit.client.0.trial112.stdout:adding osd2 key to auth repository 2026-02-13T02:01:35.801 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:01:36.002 INFO:tasks.workunit.client.0.trial112.stdout:start osd.2 2026-02-13T02:01:36.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-13T02:01:36.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-13T02:01:36.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:01:36.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:01:36.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:01:36.027 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:36.019+0000 7f8fd9e04a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:01:36.029 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:36.019+0000 7f8fd9e04a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:01:36.030 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:36.023+0000 7f8fd9e04a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:01:36.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-13T02:01:36.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:01:36.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T02:01:36.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:01:36.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:01:36.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:01:36.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:01:36.167 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:01:36.167 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:01:36.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:01:36.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:01:36.451 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:36.443+0000 7f8fd9e04a40 -1 Falling back to public interface 2026-02-13T02:01:37.050 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:37.039+0000 7f8fd9e04a40 -1 osd.2 0 log_to_monitors true 2026-02-13T02:01:37.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:01:37.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:01:37.326 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:01:37.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:01:37.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:01:37.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:01:37.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:01:38.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:01:38.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:01:38.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:01:38.481 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:01:38.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:01:38.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:01:38.643 INFO:tasks.workunit.client.0.trial112.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2527509367,v1:127.0.0.1:6819/2527509367] [v2:127.0.0.1:6820/2527509367,v1:127.0.0.1:6821/2527509367] exists,up 04e5adac-355e-48c7-a14a-582e1b533035 2026-02-13T02:01:38.644 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:01:38.644 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:01:38.644 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:01:38.645 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: auto_repair_erasure_coded: create_rbd_pool 2026-02-13T02:01:38.645 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-13T02:01:38.794 INFO:tasks.workunit.client.0.trial112.stderr:pool 'rbd' does not exist 2026-02-13T02:01:38.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-13T02:01:38.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-13T02:01:39.048 INFO:tasks.workunit.client.0.trial112.stderr:pool 'rbd' created 2026-02-13T02:01:39.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:01:40.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-13T02:01:40.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:344: auto_repair_erasure_coded: wait_for_clean 2026-02-13T02:01:40.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:01:40.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:01:40.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:01:40.341 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:01:40.342 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:01:40.342 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:01:40.342 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:01:40.342 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:01:40.342 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:01:40.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:01:40.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:01:40.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:01:40.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:01:40.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:01:40.454 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:01:40.618 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:01:40.618 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:01:40.618 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:01:40.618 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:01:40.619 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:01:40.619 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:01:40.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-13T02:01:40.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-13T02:01:40.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-13T02:01:40.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:01:40.693 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:01:40.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672963 2026-02-13T02:01:40.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672963 2026-02-13T02:01:40.764 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-42949672963' 2026-02-13T02:01:40.764 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:01:40.764 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:01:40.839 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542146 2026-02-13T02:01:40.839 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542146 2026-02-13T02:01:40.839 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-42949672963 2-60129542146' 2026-02-13T02:01:40.839 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:01:40.840 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-13T02:01:40.840 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:01:40.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:01:40.841 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-13T02:01:40.841 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:01:40.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-13T02:01:40.843 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836483 2026-02-13T02:01:40.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-13T02:01:40.844 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:01:41.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-13T02:01:41.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:01:42.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:01:42.011 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:01:42.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836483 2026-02-13T02:01:42.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:01:42.169 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672963 2026-02-13T02:01:42.169 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:01:42.170 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:01:42.171 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672963 2026-02-13T02:01:42.171 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:01:42.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672963 2026-02-13T02:01:42.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2026-02-13T02:01:42.173 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 42949672963 2026-02-13T02:01:42.173 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:01:42.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672963 -lt 42949672963 2026-02-13T02:01:42.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:01:42.346 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542146 2026-02-13T02:01:42.346 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:01:42.347 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:01:42.347 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542146 2026-02-13T02:01:42.348 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:01:42.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542146 2026-02-13T02:01:42.350 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542146 2026-02-13T02:01:42.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542146' 2026-02-13T02:01:42.350 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:01:42.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542146 -lt 60129542146 2026-02-13T02:01:42.519 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:01:42.520 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:01:42.520 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:01:42.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-13T02:01:42.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:01:42.713 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:01:42.713 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:01:42.714 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:01:42.714 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:01:42.714 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:01:42.714 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:01:42.876 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-13T02:01:42.877 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:01:42.877 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:01:42.877 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:01:43.074 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-13T02:01:43.074 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:01:43.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:01:43.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:347: auto_repair_erasure_coded: create_ec_pool ecpool true k=2 m=1 2026-02-13T02:01:43.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2522: create_ec_pool: local pool_name=ecpool 2026-02-13T02:01:43.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2523: create_ec_pool: shift 2026-02-13T02:01:43.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2524: create_ec_pool: local allow_overwrites=true 2026-02-13T02:01:43.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2525: create_ec_pool: shift 2026-02-13T02:01:43.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2527: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2026-02-13T02:01:43.278 INFO:tasks.workunit.client.0.trial112.stderr:load: isa 2026-02-13T02:01:43.290 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2529: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2026-02-13T02:01:43.291 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2026-02-13T02:01:43.583 INFO:tasks.workunit.client.0.trial112.stderr:pool 'ecpool' created 2026-02-13T02:01:43.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:01:44.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2531: create_ec_pool: '[' true = true ']' 2026-02-13T02:01:44.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2532: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2026-02-13T02:01:44.761 INFO:tasks.workunit.client.0.trial112.stderr:set pool 2 allow_ec_overwrites to true 2026-02-13T02:01:44.771 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2535: create_ec_pool: wait_for_clean 2026-02-13T02:01:44.771 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:01:44.771 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:01:44.771 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:01:44.771 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:01:44.772 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:01:44.772 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:01:44.773 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:01:44.773 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:01:44.773 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:01:44.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:01:44.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:01:44.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:01:44.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:01:44.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:01:44.888 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:01:45.052 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:01:45.053 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:01:45.053 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:01:45.053 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:01:45.053 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:01:45.053 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:01:45.129 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836485 2026-02-13T02:01:45.129 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836485 2026-02-13T02:01:45.129 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485' 2026-02-13T02:01:45.129 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:01:45.129 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:01:45.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672965 2026-02-13T02:01:45.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672965 2026-02-13T02:01:45.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-42949672965' 2026-02-13T02:01:45.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:01:45.201 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:01:45.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542148 2026-02-13T02:01:45.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542148 2026-02-13T02:01:45.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-42949672965 2-60129542148' 2026-02-13T02:01:45.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:01:45.274 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836485 2026-02-13T02:01:45.274 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:01:45.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:01:45.275 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836485 2026-02-13T02:01:45.275 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:01:45.277 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836485 2026-02-13T02:01:45.277 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836485 2026-02-13T02:01:45.277 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-13T02:01:45.278 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:01:45.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836485 2026-02-13T02:01:45.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:01:46.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:01:46.445 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:01:46.610 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 21474836485 2026-02-13T02:01:46.611 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:01:46.611 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672965 2026-02-13T02:01:46.611 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:01:46.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:01:46.613 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672965 2026-02-13T02:01:46.613 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:01:46.615 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672965 2026-02-13T02:01:46.615 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 42949672965 2026-02-13T02:01:46.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2026-02-13T02:01:46.616 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:01:46.773 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672965 -lt 42949672965 2026-02-13T02:01:46.774 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:01:46.774 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542148 2026-02-13T02:01:46.774 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:01:46.775 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:01:46.776 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542148 2026-02-13T02:01:46.776 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:01:46.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542148 2026-02-13T02:01:46.778 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542148 2026-02-13T02:01:46.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542148' 2026-02-13T02:01:46.779 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:01:46.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542148 -lt 60129542148 2026-02-13T02:01:46.941 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:01:46.941 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:01:46.941 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:01:47.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:01:47.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:01:47.136 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:01:47.136 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:01:47.136 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:01:47.136 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:01:47.136 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:01:47.136 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:01:47.291 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:01:47.292 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:01:47.292 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:01:47.292 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:01:47.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:01:47.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:01:47.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:01:47.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2536: create_ec_pool: return 0 2026-02-13T02:01:47.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:350: auto_repair_erasure_coded: local payload=ABCDEF 2026-02-13T02:01:47.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:351: auto_repair_erasure_coded: echo ABCDEF 2026-02-13T02:01:47.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-13T02:01:47.504 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:356: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2026-02-13T02:01:47.504 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=ecpool 2026-02-13T02:01:47.504 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-13T02:01:47.504 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary ecpool SOMETHING 2026-02-13T02:01:47.504 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2026-02-13T02:01:47.505 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-13T02:01:47.505 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2026-02-13T02:01:47.505 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T02:01:47.668 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=2 2026-02-13T02:01:47.668 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map ecpool SOMETHING 2026-02-13T02:01:47.669 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2026-02-13T02:01:47.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:356: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2026-02-13T02:01:47.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:01:47.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:01:47.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:01:47.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:01:47.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2026-02-13T02:01:47.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:01:47.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:01:47.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:01:47.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:01:47.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:01:47.833 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:01:47.833 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:01:47.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:01:47.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:01:47.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:01:48.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:01:48.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2026-02-13T02:01:48.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:01:48.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:01:48.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:01:48.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:01:48.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:01:48.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2026-02-13T02:01:48.337 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::deep_scrub_1.3:head#, (61) No data available 2026-02-13T02:01:48.338 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2026-02-13T02:01:48.338 INFO:tasks.workunit.client.0.trial112.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2026-02-13T02:01:48.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-13T02:01:48.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:01:48.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:01:48.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:01:48.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:01:48.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:01:48.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:01:48.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:01:48.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:01:48.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:01:48.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:01:48.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:01:48.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:01:48.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:01:48.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:01:48.426 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:01:48.426 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:01:48.426 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:01:48.426 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:01:48.427 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:01:48.427 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:01:48.427 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:01:48.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:01:48.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:01:48.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:01:48.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:01:48.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:01:48.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:01:48.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:01:48.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:01:48.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:01:48.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2026-02-13T02:01:48.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:01:48.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:01:48.429 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:01:48.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-13T02:01:48.429 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:01:48.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:01:48.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:01:48.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:01:48.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:01:48.446 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:48.439+0000 7f05431d6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:01:48.447 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:48.439+0000 7f05431d6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:01:48.448 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:48.439+0000 7f05431d6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:01:48.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:01:48.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:01:48.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:01:48.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:01:48.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:01:48.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:01:48.594 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:01:48.594 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:01:48.594 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:01:48.594 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:01:48.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:01:48.831 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:48.823+0000 7f05431d6a40 -1 Falling back to public interface 2026-02-13T02:01:49.409 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:01:49.399+0000 7f05431d6a40 -1 osd.1 27 log_to_monitors true 2026-02-13T02:01:49.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:01:49.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:01:49.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:01:49.758 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:01:49.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:01:49.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:01:49.915 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 31 up_thru 31 down_at 28 last_clean_interval [10,27) [v2:127.0.0.1:6810/2027373788,v1:127.0.0.1:6811/2027373788] [v2:127.0.0.1:6812/2027373788,v1:127.0.0.1:6813/2027373788] exists,up abae2a6d-0142-4540-a20c-ec3bdc796aaf 2026-02-13T02:01:49.915 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:01:49.915 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:01:49.915 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:01:49.915 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:01:49.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:01:49.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:01:49.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:01:49.916 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:01:49.917 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:01:49.917 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:01:49.917 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:01:49.917 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:01:49.917 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:01:50.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:01:50.021 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:01:50.021 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:01:50.021 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:01:50.021 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:01:50.021 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:01:50.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:01:50.189 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:01:50.189 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:01:50.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:01:50.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:01:50.190 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:01:50.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836487 2026-02-13T02:01:50.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836487 2026-02-13T02:01:50.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487' 2026-02-13T02:01:50.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:01:50.262 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:01:50.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986178 2026-02-13T02:01:50.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986178 2026-02-13T02:01:50.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487 1-133143986178' 2026-02-13T02:01:50.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:01:50.334 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:01:50.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542150 2026-02-13T02:01:50.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542150 2026-02-13T02:01:50.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487 1-133143986178 2-60129542150' 2026-02-13T02:01:50.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:01:50.406 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836487 2026-02-13T02:01:50.406 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:01:50.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:01:50.408 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836487 2026-02-13T02:01:50.408 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:01:50.410 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836487 2026-02-13T02:01:50.410 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2026-02-13T02:01:50.410 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 21474836487 2026-02-13T02:01:50.410 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:01:50.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836487 2026-02-13T02:01:50.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:01:51.568 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:01:51.568 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:01:51.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836487 2026-02-13T02:01:51.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:01:52.735 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:01:52.736 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:01:52.899 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836487 2026-02-13T02:01:52.899 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:01:52.900 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-133143986178 2026-02-13T02:01:52.900 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:01:52.901 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:01:52.902 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-133143986178 2026-02-13T02:01:52.902 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:01:52.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=133143986178 2026-02-13T02:01:52.904 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 133143986178' 2026-02-13T02:01:52.904 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 133143986178 2026-02-13T02:01:52.904 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:01:53.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 133143986178 -lt 133143986178 2026-02-13T02:01:53.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:01:53.070 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542150 2026-02-13T02:01:53.070 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:01:53.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:01:53.072 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542150 2026-02-13T02:01:53.072 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:01:53.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542150 2026-02-13T02:01:53.074 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542150' 2026-02-13T02:01:53.074 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 60129542150 2026-02-13T02:01:53.074 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:01:53.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542150 -lt 60129542150 2026-02-13T02:01:53.246 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:01:53.246 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:01:53.246 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:01:53.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:01:53.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:01:53.448 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:01:53.448 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:01:53.449 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:01:53.449 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:01:53.449 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:01:53.449 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:01:53.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:01:53.609 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:01:53.609 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:01:53.609 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:01:53.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:01:53.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:01:53.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:01:53.809 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:358: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2026-02-13T02:01:53.809 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-13T02:01:53.809 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-13T02:01:53.809 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-13T02:01:53.810 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T02:01:53.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:358: auto_repair_erasure_coded: local pgid=2.0 2026-02-13T02:01:53.976 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:359: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2026-02-13T02:01:53.976 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:01:53.976 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:01:53.976 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:01:53.976 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:01:54.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:359: auto_repair_erasure_coded: wait_for_scrub 2.0 2026-02-13T02:01:43.579332+0000 2026-02-13T02:01:54.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-13T02:01:54.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:01:43.579332+0000 2026-02-13T02:01:54.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:01:54.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:01:54.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:01:54.140 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:01:54.141 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:01:54.141 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:01:54.141 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:01:54.141 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:01:54.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:01:43.579332+0000 '>' 2026-02-13T02:01:43.579332+0000 2026-02-13T02:01:54.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:01:55.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:01:55.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:01:55.306 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:01:55.306 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:01:55.306 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:01:55.307 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:01:55.307 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:01:55.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:01:43.579332+0000 '>' 2026-02-13T02:01:43.579332+0000 2026-02-13T02:01:55.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:01:56.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:01:56.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:01:56.469 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:01:56.469 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:01:56.469 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:01:56.469 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:01:56.469 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:01:56.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:01:43.579332+0000 '>' 2026-02-13T02:01:43.579332+0000 2026-02-13T02:01:56.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:01:57.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:01:57.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:01:57.633 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:01:57.633 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:01:57.633 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:01:57.633 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:01:57.633 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:01:57.795 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:01:43.579332+0000 '>' 2026-02-13T02:01:43.579332+0000 2026-02-13T02:01:57.795 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:01:58.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:01:58.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:01:58.797 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:01:58.797 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:01:58.798 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:01:58.798 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:01:58.798 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:01:58.959 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:01:43.579332+0000 '>' 2026-02-13T02:01:43.579332+0000 2026-02-13T02:01:58.959 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:01:59.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:01:59.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:01:59.961 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:01:59.962 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:01:59.962 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:01:59.962 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:01:59.962 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:02:00.122 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:01:57.859201+0000 '>' 2026-02-13T02:01:43.579332+0000 2026-02-13T02:02:00.122 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:02:00.122 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:360: auto_repair_erasure_coded: wait_for_clean 2026-02-13T02:02:00.123 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:02:00.123 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:02:00.123 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:02:00.123 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:02:00.124 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:02:00.124 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:02:00.124 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:02:00.124 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:02:00.124 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:02:00.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:02:00.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:02:00.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:02:00.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:02:00.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:02:00.226 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:02:00.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:02:00.386 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:02:00.386 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:02:00.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:02:00.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:02:00.387 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:02:00.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836490 2026-02-13T02:02:00.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836490 2026-02-13T02:02:00.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836490' 2026-02-13T02:02:00.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:02:00.460 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:02:00.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986181 2026-02-13T02:02:00.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986181 2026-02-13T02:02:00.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836490 1-133143986181' 2026-02-13T02:02:00.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:02:00.535 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:02:00.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542153 2026-02-13T02:02:00.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542153 2026-02-13T02:02:00.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836490 1-133143986181 2-60129542153' 2026-02-13T02:02:00.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:02:00.606 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836490 2026-02-13T02:02:00.606 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:02:00.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:02:00.608 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836490 2026-02-13T02:02:00.608 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:02:00.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836490 2026-02-13T02:02:00.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2026-02-13T02:02:00.609 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836490 2026-02-13T02:02:00.610 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:02:00.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836490 2026-02-13T02:02:00.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:02:01.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:02:01.782 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:02:01.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836490 2026-02-13T02:02:01.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:02:02.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:02:02.943 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:02:03.105 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836490 2026-02-13T02:02:03.105 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:02:03.106 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-133143986181 2026-02-13T02:02:03.106 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:02:03.107 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:02:03.108 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-133143986181 2026-02-13T02:02:03.108 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:02:03.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=133143986181 2026-02-13T02:02:03.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 133143986181' 2026-02-13T02:02:03.110 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 133143986181 2026-02-13T02:02:03.110 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:02:03.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 133143986181 -lt 133143986181 2026-02-13T02:02:03.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:02:03.280 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542153 2026-02-13T02:02:03.280 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:02:03.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:02:03.281 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542153 2026-02-13T02:02:03.282 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:02:03.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542153 2026-02-13T02:02:03.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542153' 2026-02-13T02:02:03.283 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542153 2026-02-13T02:02:03.284 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:02:03.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542153 -lt 60129542153 2026-02-13T02:02:03.446 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:02:03.446 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:02:03.446 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:02:03.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:02:03.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:02:03.638 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:02:03.638 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:02:03.639 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:02:03.639 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:02:03.639 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:02:03.639 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:02:03.795 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:02:03.796 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:02:03.796 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:02:03.796 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:02:03.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:02:03.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:02:03.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:02:03.987 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2026-02-13T02:02:03.987 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=ecpool 2026-02-13T02:02:03.988 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-13T02:02:03.988 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary ecpool SOMETHING 2026-02-13T02:02:03.988 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2026-02-13T02:02:03.988 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-13T02:02:03.988 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2026-02-13T02:02:03.988 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T02:02:04.148 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=2 2026-02-13T02:02:04.149 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map ecpool SOMETHING 2026-02-13T02:02:04.149 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2026-02-13T02:02:04.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-13T02:02:04.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:02:04.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:02:04.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:02:04.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:02:04.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-13T02:02:04.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:02:04.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:02:04.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:02:04.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:02:04.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:02:04.314 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:02:04.314 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:02:04.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:02:04.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:02:04.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:02:04.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:02:04.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-13T02:02:04.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:02:04.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:02:04.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:02:04.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:02:04.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:02:04.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2026-02-13T02:02:04.688 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::deep_scrub_1.3:head#, (61) No data available 2026-02-13T02:02:04.688 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2026-02-13T02:02:04.688 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.0_head,#-3:00000000:::deep_scrub_1.0:head#, (61) No data available 2026-02-13T02:02:04.688 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2026-02-13T02:02:04.688 INFO:tasks.workunit.client.0.trial112.stdout:_ 2026-02-13T02:02:04.689 INFO:tasks.workunit.client.0.trial112.stdout:hinfo_key 2026-02-13T02:02:04.689 INFO:tasks.workunit.client.0.trial112.stdout:snapset 2026-02-13T02:02:04.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-13T02:02:04.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:02:04.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:02:04.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:02:04.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:02:04.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:02:04.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:02:04.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:02:04.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:02:04.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:02:04.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:02:04.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:02:04.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:02:04.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:02:04.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:02:04.778 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:02:04.778 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:02:04.778 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:02:04.778 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:02:04.778 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:02:04.778 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:02:04.778 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:02:04.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:02:04.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:02:04.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:02:04.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:02:04.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:02:04.779 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:02:04.779 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:02:04.779 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:02:04.779 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:02:04.779 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2026-02-13T02:02:04.779 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:02:04.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:02:04.780 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:02:04.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-13T02:02:04.780 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:02:04.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:02:04.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:02:04.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:02:04.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:02:04.798 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:04.790+0000 7fdbfb835a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:02:04.799 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:04.794+0000 7fdbfb835a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:02:04.800 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:04.794+0000 7fdbfb835a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:02:04.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:02:04.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:02:04.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:02:04.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:02:04.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:02:04.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:02:04.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:02:04.941 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:02:04.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:02:04.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:02:05.105 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:02:05.136 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:05.130+0000 7fdbfb835a40 -1 Falling back to public interface 2026-02-13T02:02:05.747 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:05.742+0000 7fdbfb835a40 -1 osd.1 33 log_to_monitors true 2026-02-13T02:02:06.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:02:06.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:02:06.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:02:06.106 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:02:06.107 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:02:06.107 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:02:06.266 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 37 up_thru 37 down_at 34 last_clean_interval [31,33) [v2:127.0.0.1:6810/942988975,v1:127.0.0.1:6811/942988975] [v2:127.0.0.1:6812/942988975,v1:127.0.0.1:6813/942988975] exists,up abae2a6d-0142-4540-a20c-ec3bdc796aaf 2026-02-13T02:02:06.266 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:02:06.266 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:02:06.266 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:02:06.266 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:02:06.266 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:02:06.266 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:02:06.267 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:02:06.267 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:02:06.268 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:02:06.268 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:02:06.268 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:02:06.268 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:02:06.268 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:02:06.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:02:06.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:02:06.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:02:06.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:02:06.368 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:02:06.368 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:02:06.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:02:06.529 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:02:06.529 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:02:06.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:02:06.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:02:06.529 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:02:06.601 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836493 2026-02-13T02:02:06.601 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836493 2026-02-13T02:02:06.601 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493' 2026-02-13T02:02:06.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:02:06.602 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:02:06.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789954 2026-02-13T02:02:06.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789954 2026-02-13T02:02:06.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-158913789954' 2026-02-13T02:02:06.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:02:06.676 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:02:06.748 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542155 2026-02-13T02:02:06.748 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542155 2026-02-13T02:02:06.748 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-158913789954 2-60129542155' 2026-02-13T02:02:06.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:02:06.749 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836493 2026-02-13T02:02:06.749 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:02:06.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:02:06.751 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836493 2026-02-13T02:02:06.751 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:02:06.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836493 2026-02-13T02:02:06.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2026-02-13T02:02:06.753 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 21474836493 2026-02-13T02:02:06.753 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:02:06.926 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836493 2026-02-13T02:02:06.926 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:02:07.927 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:02:07.928 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:02:08.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836493 2026-02-13T02:02:08.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:02:09.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:02:09.086 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:02:09.251 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836493 -lt 21474836493 2026-02-13T02:02:09.251 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:02:09.252 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-158913789954 2026-02-13T02:02:09.252 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:02:09.253 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:02:09.254 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-158913789954 2026-02-13T02:02:09.254 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:02:09.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789954 2026-02-13T02:02:09.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 158913789954' 2026-02-13T02:02:09.256 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 158913789954 2026-02-13T02:02:09.256 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:02:09.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789954 -lt 158913789954 2026-02-13T02:02:09.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:02:09.412 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542155 2026-02-13T02:02:09.412 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:02:09.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:02:09.414 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542155 2026-02-13T02:02:09.414 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:02:09.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542155 2026-02-13T02:02:09.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542155' 2026-02-13T02:02:09.416 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 60129542155 2026-02-13T02:02:09.416 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:02:09.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542155 -lt 60129542155 2026-02-13T02:02:09.580 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:02:09.580 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:02:09.580 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:02:09.773 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:02:09.773 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:02:09.774 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:02:09.775 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:02:09.775 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:02:09.775 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:02:09.775 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:02:09.775 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:02:09.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:02:09.934 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:02:09.934 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:02:09.934 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:02:10.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:02:10.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:02:10.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:02:10.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-13T02:02:10.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-13T02:02:10.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-13T02:02:10.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:02:10.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:02:10.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:02:10.148 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:02:10.148 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:02:10.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:02:10.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:02:10.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:02:10.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:02:10.258 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:02:10.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:02:10.259 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:02:10.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:02:10.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:02:10.261 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:02:10.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:02:10.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:02:10.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:02:10.263 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:02:10.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:02:10.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:02:10.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:02:10.280 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:02:10.280 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:02:10.280 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:02:10.280 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:02:10.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:02:10.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:02:10.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-13T02:02:10.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-13T02:02:10.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-13T02:02:10.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-13T02:02:10.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:02:10.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:02:10.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:02:10.283 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:02:10.283 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:02:10.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:02:10.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:02:10.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:02:10.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:02:10.285 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:02:10.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:02:10.286 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:02:10.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:02:10.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:02:10.288 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:02:10.290 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:02:10.290 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:02:10.290 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:02:10.290 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:02:10.291 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:02:10.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:02:10.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:02:10.294 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:02:10.294 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:02:10.294 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:02:10.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:02:10.295 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:02:10.295 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:02:10.295 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-13T02:02:10.297 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T02:02:10.297 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:02:10.297 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:02:10.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.35207 2026-02-13T02:02:10.299 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T02:02:10.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-13T02:02:10.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T02:02:10.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-13T02:02:10.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_jerasure_appends td/osd-scrub-repair 2026-02-13T02:02:10.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:874: TEST_corrupt_and_repair_jerasure_appends: corrupt_and_repair_jerasure td/osd-scrub-repair false 2026-02-13T02:02:10.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:857: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2026-02-13T02:02:10.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:858: corrupt_and_repair_jerasure: local allow_overwrites=false 2026-02-13T02:02:10.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: corrupt_and_repair_jerasure: local poolname=ecpool 2026-02-13T02:02:10.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:861: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2026-02-13T02:02:10.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-13T02:02:10.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-13T02:02:10.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-13T02:02:10.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-13T02:02:10.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-13T02:02:10.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-13T02:02:10.322 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-13T02:02:10.322 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:02:10.322 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:02:10.322 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:02:10.322 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:02:10.322 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:02:10.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:02:10.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-13T02:02:10.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-13T02:02:10.351 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-13T02:02:10.351 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:02:10.351 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:02:10.351 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-13T02:02:10.351 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-13T02:02:10.353 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:02:10.353 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:02:10.353 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:02:10.353 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:02:10.353 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:02:10.353 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:02:10.353 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:02:10.353 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:02:10.353 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get fsid 2026-02-13T02:02:10.411 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-13T02:02:10.411 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:02:10.412 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:02:10.412 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-13T02:02:10.412 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-13T02:02:10.412 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:02:10.412 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:02:10.412 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:02:10.413 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:02:10.413 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:02:10.413 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:02:10.413 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:02:10.414 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:02:10.414 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get mon_host 2026-02-13T02:02:10.469 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:862: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2026-02-13T02:02:10.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-13T02:02:10.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-13T02:02:10.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-13T02:02:10.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-13T02:02:10.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-13T02:02:10.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-13T02:02:10.565 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-13T02:02:10.565 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:02:10.565 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:02:10.566 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:02:10.566 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:02:10.566 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:02:10.566 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:02:10.567 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:02:10.568 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:02:10.588 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: corrupt_and_repair_jerasure: seq 0 3 2026-02-13T02:02:10.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2026-02-13T02:02:10.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:864: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2026-02-13T02:02:10.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:02:10.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:02:10.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-13T02:02:10.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:02:10.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:02:10.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:02:10.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:02:10.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:02:10.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:02:10.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:02:10.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:02:10.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:02:10.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:02:10.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:02:10.592 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:02:10.592 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:02:10.592 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:02:10.592 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:02:10.592 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:02:10.592 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:02:10.592 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:02:10.594 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:02:10.594 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:02:10.594 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:02:10.594 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:02:10.594 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:02:10.594 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:02:10.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:02:10.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:02:10.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:02:10.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:02:10.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:02:10.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:02:10.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:02:10.595 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:02:10.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d8b42ef2-111b-4754-9924-5a83049d3790 2026-02-13T02:02:10.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 d8b42ef2-111b-4754-9924-5a83049d3790' 2026-02-13T02:02:10.596 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 d8b42ef2-111b-4754-9924-5a83049d3790 2026-02-13T02:02:10.596 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:02:10.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAiho5p8HPOIxAAGctNkejFvFKAqXQ4hw0B+Q== 2026-02-13T02:02:10.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAiho5p8HPOIxAAGctNkejFvFKAqXQ4hw0B+Q=="}' 2026-02-13T02:02:10.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d8b42ef2-111b-4754-9924-5a83049d3790 -i td/osd-scrub-repair/0/new.json 2026-02-13T02:02:10.695 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:02:10.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-13T02:02:10.705 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAiho5p8HPOIxAAGctNkejFvFKAqXQ4hw0B+Q== --osd-uuid d8b42ef2-111b-4754-9924-5a83049d3790 2026-02-13T02:02:10.720 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:10.714+0000 7f67d3a01a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:02:10.721 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:10.714+0000 7f67d3a01a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:02:10.722 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:10.718+0000 7f67d3a01a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:02:10.723 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:10.718+0000 7f67d3a01a40 -1 bdev(0x563f8355ec00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-13T02:02:10.723 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:10.718+0000 7f67d3a01a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-13T02:02:11.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-13T02:02:11.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:02:11.447 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T02:02:11.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-13T02:02:11.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:02:11.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-13T02:02:11.542 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T02:02:11.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:02:11.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:02:11.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:02:11.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:02:11.557 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:11.550+0000 7fcea3531a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:02:11.558 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:11.554+0000 7fcea3531a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:02:11.559 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:11.554+0000 7fcea3531a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:02:11.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-13T02:02:11.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:02:11.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:02:11.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:02:11.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:02:11.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:02:11.638 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:02:11.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:02:11.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:02:11.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:02:11.729 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:02:11.980 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:11.974+0000 7fcea3531a40 -1 Falling back to public interface 2026-02-13T02:02:12.608 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:12.602+0000 7fcea3531a40 -1 osd.0 0 log_to_monitors true 2026-02-13T02:02:12.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:02:12.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:02:12.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:02:12.732 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:02:12.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:02:12.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:02:12.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:02:13.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:02:13.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:02:13.862 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:02:13.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:02:13.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:02:13.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:02:14.022 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2889230893,v1:127.0.0.1:6801/2889230893] [v2:127.0.0.1:6802/2889230893,v1:127.0.0.1:6803/2889230893] exists,up d8b42ef2-111b-4754-9924-5a83049d3790 2026-02-13T02:02:14.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:02:14.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:02:14.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:02:14.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2026-02-13T02:02:14.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:864: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2026-02-13T02:02:14.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:02:14.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:02:14.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-13T02:02:14.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:02:14.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:02:14.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:02:14.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:02:14.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:02:14.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:02:14.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:02:14.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:02:14.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:02:14.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:02:14.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:02:14.027 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:02:14.028 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:02:14.028 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:02:14.028 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:02:14.028 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:02:14.028 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:02:14.028 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:02:14.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:02:14.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:02:14.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:02:14.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:02:14.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:02:14.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:02:14.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:02:14.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:02:14.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:02:14.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:02:14.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:02:14.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:02:14.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:02:14.030 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:02:14.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=776209f2-682b-43d3-a00a-d3186f312ec7 2026-02-13T02:02:14.031 INFO:tasks.workunit.client.0.trial112.stdout:add osd1 776209f2-682b-43d3-a00a-d3186f312ec7 2026-02-13T02:02:14.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 776209f2-682b-43d3-a00a-d3186f312ec7' 2026-02-13T02:02:14.032 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:02:14.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAmho5pW5IbAhAAZGEj0fgfkfisxWgV6O3vrQ== 2026-02-13T02:02:14.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAmho5pW5IbAhAAZGEj0fgfkfisxWgV6O3vrQ=="}' 2026-02-13T02:02:14.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 776209f2-682b-43d3-a00a-d3186f312ec7 -i td/osd-scrub-repair/1/new.json 2026-02-13T02:02:14.205 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:02:14.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-13T02:02:14.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAmho5pW5IbAhAAZGEj0fgfkfisxWgV6O3vrQ== --osd-uuid 776209f2-682b-43d3-a00a-d3186f312ec7 2026-02-13T02:02:14.231 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:14.226+0000 7f3e964dca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:02:14.233 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:14.226+0000 7f3e964dca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:02:14.234 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:14.226+0000 7f3e964dca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:02:14.234 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:14.230+0000 7f3e964dca40 -1 bdev(0x562e8fed4c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-13T02:02:14.235 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:14.230+0000 7f3e964dca40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-13T02:02:14.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-13T02:02:14.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:02:14.890 INFO:tasks.workunit.client.0.trial112.stdout:adding osd1 key to auth repository 2026-02-13T02:02:14.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-13T02:02:14.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:02:15.093 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T02:02:15.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-13T02:02:15.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:02:15.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:02:15.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:02:15.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:02:15.109 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:15.102+0000 7f4ce087ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:02:15.110 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:15.102+0000 7f4ce087ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:02:15.111 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:15.106+0000 7f4ce087ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:02:15.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-13T02:02:15.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:02:15.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:02:15.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:02:15.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:02:15.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:02:15.253 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:02:15.253 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:02:15.253 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:02:15.254 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:02:15.396 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:15.390+0000 7f4ce087ca40 -1 Falling back to public interface 2026-02-13T02:02:15.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:02:16.028 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:16.022+0000 7f4ce087ca40 -1 osd.1 0 log_to_monitors true 2026-02-13T02:02:16.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:02:16.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:02:16.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:02:16.420 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:02:16.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:02:16.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:02:16.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:02:17.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:02:17.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:02:17.579 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:02:17.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:02:17.580 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:02:17.580 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:02:17.741 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/534173962,v1:127.0.0.1:6811/534173962] [v2:127.0.0.1:6812/534173962,v1:127.0.0.1:6813/534173962] exists,up 776209f2-682b-43d3-a00a-d3186f312ec7 2026-02-13T02:02:17.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:02:17.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:02:17.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:02:17.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2026-02-13T02:02:17.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:864: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2026-02-13T02:02:17.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:02:17.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:02:17.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-13T02:02:17.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:02:17.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-13T02:02:17.747 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:02:17.747 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:02:17.747 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:02:17.747 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:02:17.747 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-13T02:02:17.747 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-13T02:02:17.748 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:02:17.748 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:02:17.748 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:02:17.748 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:02:17.748 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:02:17.748 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:02:17.748 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:02:17.748 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:02:17.748 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:02:17.748 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:02:17.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:02:17.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:02:17.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:02:17.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:02:17.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:02:17.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:02:17.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:02:17.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:02:17.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:02:17.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:02:17.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:02:17.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:02:17.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-13T02:02:17.750 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:02:17.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6d9c196a-4cd8-413e-bba5-8ecb673833b9 2026-02-13T02:02:17.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 6d9c196a-4cd8-413e-bba5-8ecb673833b9' 2026-02-13T02:02:17.751 INFO:tasks.workunit.client.0.trial112.stdout:add osd2 6d9c196a-4cd8-413e-bba5-8ecb673833b9 2026-02-13T02:02:17.751 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:02:17.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQApho5pFDnnLBAAnSgaK1zo458jlXtrIO0WXg== 2026-02-13T02:02:17.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQApho5pFDnnLBAAnSgaK1zo458jlXtrIO0WXg=="}' 2026-02-13T02:02:17.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6d9c196a-4cd8-413e-bba5-8ecb673833b9 -i td/osd-scrub-repair/2/new.json 2026-02-13T02:02:17.927 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:02:17.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-13T02:02:17.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQApho5pFDnnLBAAnSgaK1zo458jlXtrIO0WXg== --osd-uuid 6d9c196a-4cd8-413e-bba5-8ecb673833b9 2026-02-13T02:02:17.955 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:17.950+0000 7ff66908aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:02:17.956 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:17.950+0000 7ff66908aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:02:17.957 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:17.950+0000 7ff66908aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:02:17.958 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:17.950+0000 7ff66908aa40 -1 bdev(0x5579db6eac00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-13T02:02:17.958 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:17.950+0000 7ff66908aa40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-13T02:02:18.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-13T02:02:18.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:02:18.590 INFO:tasks.workunit.client.0.trial112.stdout:adding osd2 key to auth repository 2026-02-13T02:02:18.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-13T02:02:18.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:02:18.791 INFO:tasks.workunit.client.0.trial112.stdout:start osd.2 2026-02-13T02:02:18.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-13T02:02:18.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:02:18.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:02:18.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:02:18.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:02:18.807 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:18.802+0000 7f8ebc8c4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:02:18.809 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:18.802+0000 7f8ebc8c4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:02:18.810 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:18.802+0000 7f8ebc8c4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:02:18.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-13T02:02:18.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:02:18.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T02:02:18.950 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:02:18.950 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:02:18.950 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:02:18.950 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:02:18.950 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:02:18.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:02:18.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:02:19.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:02:19.232 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:19.226+0000 7f8ebc8c4a40 -1 Falling back to public interface 2026-02-13T02:02:19.800 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:19.794+0000 7f8ebc8c4a40 -1 osd.2 0 log_to_monitors true 2026-02-13T02:02:20.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:02:20.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:02:20.115 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:02:20.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:02:20.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:02:20.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:02:20.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:02:21.280 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:02:21.280 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:02:21.280 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:02:21.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:02:21.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:02:21.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:02:21.444 INFO:tasks.workunit.client.0.trial112.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1840045323,v1:127.0.0.1:6819/1840045323] [v2:127.0.0.1:6820/1840045323,v1:127.0.0.1:6821/1840045323] exists,up 6d9c196a-4cd8-413e-bba5-8ecb673833b9 2026-02-13T02:02:21.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:02:21.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:02:21.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:02:21.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2026-02-13T02:02:21.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:864: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2026-02-13T02:02:21.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:02:21.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:02:21.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-13T02:02:21.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:02:21.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:02:21.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:02:21.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:02:21.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:02:21.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:02:21.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-13T02:02:21.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-13T02:02:21.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:02:21.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:02:21.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:02:21.450 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:02:21.450 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:02:21.450 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:02:21.450 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:02:21.450 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:02:21.450 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:02:21.450 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:02:21.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:02:21.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:02:21.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:02:21.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:02:21.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:02:21.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:02:21.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:02:21.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:02:21.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:02:21.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:02:21.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:02:21.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:02:21.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/3 2026-02-13T02:02:21.452 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:02:21.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=61ebaa55-733a-4e41-b1bf-de710ed831df 2026-02-13T02:02:21.453 INFO:tasks.workunit.client.0.trial112.stdout:add osd3 61ebaa55-733a-4e41-b1bf-de710ed831df 2026-02-13T02:02:21.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 61ebaa55-733a-4e41-b1bf-de710ed831df' 2026-02-13T02:02:21.453 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:02:21.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAtho5p1EQ5GxAAAT/aHHZzHpWzXILLlZKPjg== 2026-02-13T02:02:21.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAtho5p1EQ5GxAAAT/aHHZzHpWzXILLlZKPjg=="}' 2026-02-13T02:02:21.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 61ebaa55-733a-4e41-b1bf-de710ed831df -i td/osd-scrub-repair/3/new.json 2026-02-13T02:02:21.626 INFO:tasks.workunit.client.0.trial112.stdout:3 2026-02-13T02:02:21.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/3/new.json 2026-02-13T02:02:21.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAtho5p1EQ5GxAAAT/aHHZzHpWzXILLlZKPjg== --osd-uuid 61ebaa55-733a-4e41-b1bf-de710ed831df 2026-02-13T02:02:21.651 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:21.646+0000 7f788c785a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:02:21.653 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:21.646+0000 7f788c785a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:02:21.654 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:21.646+0000 7f788c785a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:02:21.655 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:21.650+0000 7f788c785a40 -1 bdev(0x55808231cc00 td/osd-scrub-repair/3/block) open stat got: (1) Operation not permitted 2026-02-13T02:02:21.655 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:21.650+0000 7f788c785a40 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2026-02-13T02:02:22.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2026-02-13T02:02:22.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:02:22.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-13T02:02:22.279 INFO:tasks.workunit.client.0.trial112.stdout:adding osd3 key to auth repository 2026-02-13T02:02:22.280 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:02:22.485 INFO:tasks.workunit.client.0.trial112.stdout:start osd.3 2026-02-13T02:02:22.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-13T02:02:22.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:02:22.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:02:22.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:02:22.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:02:22.509 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:22.502+0000 7fc5e004aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:02:22.511 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:22.506+0000 7fc5e004aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:02:22.512 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:22.506+0000 7fc5e004aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:02:22.643 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-13T02:02:22.643 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:02:22.643 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-13T02:02:22.643 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:02:22.643 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:02:22.643 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:02:22.644 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:02:22.645 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:02:22.645 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:02:22.645 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:02:22.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:02:22.920 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:22.914+0000 7fc5e004aa40 -1 Falling back to public interface 2026-02-13T02:02:23.505 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:23.498+0000 7fc5e004aa40 -1 osd.3 0 log_to_monitors true 2026-02-13T02:02:23.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:02:23.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:02:23.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:02:23.806 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:02:23.806 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:02:23.806 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:02:23.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:02:24.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:02:24.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:02:24.964 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:02:24.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:02:24.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:02:24.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:02:25.125 INFO:tasks.workunit.client.0.trial112.stdout:osd.3 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1051292834,v1:127.0.0.1:6827/1051292834] [v2:127.0.0.1:6828/1051292834,v1:127.0.0.1:6829/1051292834] exists,up 61ebaa55-733a-4e41-b1bf-de710ed831df 2026-02-13T02:02:25.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:02:25.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:02:25.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:02:25.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:866: corrupt_and_repair_jerasure: create_rbd_pool 2026-02-13T02:02:25.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-13T02:02:25.281 INFO:tasks.workunit.client.0.trial112.stderr:pool 'rbd' does not exist 2026-02-13T02:02:25.290 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-13T02:02:25.291 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-13T02:02:25.516 INFO:tasks.workunit.client.0.trial112.stderr:pool 'rbd' created 2026-02-13T02:02:25.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:02:26.527 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-13T02:02:26.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:867: corrupt_and_repair_jerasure: wait_for_clean 2026-02-13T02:02:26.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:02:26.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:02:26.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:02:26.808 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:02:26.810 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:02:26.810 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:02:26.810 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:02:26.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:02:26.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:02:26.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:02:26.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:02:26.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:02:26.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:02:26.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:02:26.912 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:02:27.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:02:27.076 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:02:27.076 INFO:tasks.workunit.client.0.trial112.stderr:2 2026-02-13T02:02:27.076 INFO:tasks.workunit.client.0.trial112.stderr:3' 2026-02-13T02:02:27.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:02:27.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:02:27.077 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:02:27.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836484 2026-02-13T02:02:27.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836484 2026-02-13T02:02:27.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484' 2026-02-13T02:02:27.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:02:27.150 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:02:27.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705668 2026-02-13T02:02:27.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705668 2026-02-13T02:02:27.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-38654705668' 2026-02-13T02:02:27.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:02:27.225 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:02:27.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542147 2026-02-13T02:02:27.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542147 2026-02-13T02:02:27.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-38654705668 2-60129542147' 2026-02-13T02:02:27.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:02:27.298 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:02:27.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411330 2026-02-13T02:02:27.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411330 2026-02-13T02:02:27.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-38654705668 2-60129542147 3-77309411330' 2026-02-13T02:02:27.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:02:27.371 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836484 2026-02-13T02:02:27.371 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:02:27.372 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:02:27.372 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836484 2026-02-13T02:02:27.372 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:02:27.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836484 2026-02-13T02:02:27.374 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836484 2026-02-13T02:02:27.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-13T02:02:27.375 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:02:27.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836484 2026-02-13T02:02:27.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:02:28.547 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:02:28.548 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:02:28.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836484 2026-02-13T02:02:28.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:02:29.707 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:02:29.708 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:02:29.870 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836484 2026-02-13T02:02:29.870 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:02:29.871 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705668 2026-02-13T02:02:29.871 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:02:29.872 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:02:29.873 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705668 2026-02-13T02:02:29.873 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:02:29.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705668 2026-02-13T02:02:29.875 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705668 2026-02-13T02:02:29.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705668' 2026-02-13T02:02:29.875 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:02:30.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705668 -lt 38654705668 2026-02-13T02:02:30.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:02:30.035 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542147 2026-02-13T02:02:30.035 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:02:30.037 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:02:30.037 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542147 2026-02-13T02:02:30.037 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:02:30.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542147 2026-02-13T02:02:30.039 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542147 2026-02-13T02:02:30.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542147' 2026-02-13T02:02:30.040 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:02:30.206 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542147 -lt 60129542147 2026-02-13T02:02:30.206 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:02:30.206 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-77309411330 2026-02-13T02:02:30.207 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:02:30.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:02:30.208 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-77309411330 2026-02-13T02:02:30.208 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:02:30.210 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=77309411330 2026-02-13T02:02:30.210 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.3 seq 77309411330 2026-02-13T02:02:30.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 77309411330' 2026-02-13T02:02:30.211 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:02:30.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 77309411330 -lt 77309411330 2026-02-13T02:02:30.380 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:02:30.381 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:02:30.381 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:02:30.575 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-13T02:02:30.575 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:02:30.575 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:02:30.575 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:02:30.576 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:02:30.576 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:02:30.576 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:02:30.576 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:02:30.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-13T02:02:30.732 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:02:30.733 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:02:30.733 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:02:30.924 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-13T02:02:30.925 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:02:30.925 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:02:30.925 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:869: corrupt_and_repair_jerasure: create_ec_pool ecpool false k=2 m=2 2026-02-13T02:02:30.925 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2522: create_ec_pool: local pool_name=ecpool 2026-02-13T02:02:30.925 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2523: create_ec_pool: shift 2026-02-13T02:02:30.925 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2524: create_ec_pool: local allow_overwrites=false 2026-02-13T02:02:30.925 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2525: create_ec_pool: shift 2026-02-13T02:02:30.925 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2527: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2026-02-13T02:02:31.136 INFO:tasks.workunit.client.0.trial112.stderr:load: isa 2026-02-13T02:02:31.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2529: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2026-02-13T02:02:31.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2026-02-13T02:02:31.440 INFO:tasks.workunit.client.0.trial112.stderr:pool 'ecpool' created 2026-02-13T02:02:31.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:02:32.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2531: create_ec_pool: '[' false = true ']' 2026-02-13T02:02:32.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2535: create_ec_pool: wait_for_clean 2026-02-13T02:02:32.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:02:32.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:02:32.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:02:32.454 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:02:32.455 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:02:32.455 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:02:32.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:02:32.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:02:32.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:02:32.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:02:32.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:02:32.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:02:32.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:02:32.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:02:32.561 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:02:32.718 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:02:32.718 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:02:32.719 INFO:tasks.workunit.client.0.trial112.stderr:2 2026-02-13T02:02:32.719 INFO:tasks.workunit.client.0.trial112.stderr:3' 2026-02-13T02:02:32.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:02:32.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:02:32.719 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:02:32.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836486 2026-02-13T02:02:32.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836486 2026-02-13T02:02:32.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486' 2026-02-13T02:02:32.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:02:32.794 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:02:32.866 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705670 2026-02-13T02:02:32.866 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705670 2026-02-13T02:02:32.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486 1-38654705670' 2026-02-13T02:02:32.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:02:32.867 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:02:32.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542149 2026-02-13T02:02:32.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542149 2026-02-13T02:02:32.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486 1-38654705670 2-60129542149' 2026-02-13T02:02:32.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:02:32.940 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:02:33.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411332 2026-02-13T02:02:33.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411332 2026-02-13T02:02:33.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486 1-38654705670 2-60129542149 3-77309411332' 2026-02-13T02:02:33.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:02:33.011 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836486 2026-02-13T02:02:33.011 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:02:33.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:02:33.013 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836486 2026-02-13T02:02:33.013 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:02:33.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836486 2026-02-13T02:02:33.015 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2026-02-13T02:02:33.015 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836486 2026-02-13T02:02:33.015 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:02:33.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836486 2026-02-13T02:02:33.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:02:33.171 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705670 2026-02-13T02:02:33.172 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:02:33.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:02:33.173 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705670 2026-02-13T02:02:33.174 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:02:33.175 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705670 2026-02-13T02:02:33.175 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705670' 2026-02-13T02:02:33.176 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705670 2026-02-13T02:02:33.176 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:02:33.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705670 -lt 38654705670 2026-02-13T02:02:33.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:02:33.346 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542149 2026-02-13T02:02:33.346 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:02:33.348 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:02:33.348 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542149 2026-02-13T02:02:33.349 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:02:33.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542149 2026-02-13T02:02:33.350 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542149 2026-02-13T02:02:33.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542149' 2026-02-13T02:02:33.351 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:02:33.523 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542148 -lt 60129542149 2026-02-13T02:02:33.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:02:34.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:02:34.525 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:02:34.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542148 -lt 60129542149 2026-02-13T02:02:34.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:02:35.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:02:35.689 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:02:35.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542149 -lt 60129542149 2026-02-13T02:02:35.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:02:35.852 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-77309411332 2026-02-13T02:02:35.852 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:02:35.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:02:35.854 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-77309411332 2026-02-13T02:02:35.854 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:02:35.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=77309411332 2026-02-13T02:02:35.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 77309411332' 2026-02-13T02:02:35.856 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.3 seq 77309411332 2026-02-13T02:02:35.857 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:02:36.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 77309411333 -lt 77309411332 2026-02-13T02:02:36.015 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:02:36.016 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:02:36.016 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:02:36.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:02:36.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:02:36.214 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:02:36.214 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:02:36.214 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:02:36.214 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:02:36.214 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:02:36.214 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:02:36.369 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:02:36.370 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:02:36.370 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:02:36.370 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:02:36.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:02:36.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:02:36.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:02:36.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2536: create_ec_pool: return 0 2026-02-13T02:02:36.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:870: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2026-02-13T02:02:36.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:308: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2026-02-13T02:02:36.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:309: corrupt_and_repair_erasure_coded: local poolname=ecpool 2026-02-13T02:02:36.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:311: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2026-02-13T02:02:36.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:02:36.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-13T02:02:36.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2026-02-13T02:02:36.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:02:36.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:02:36.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:02:36.794 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:02:36.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:02:37.000 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:02:37.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:02:37.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:02:37.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-13T02:02:37.038 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:313: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2026-02-13T02:02:37.039 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2026-02-13T02:02:37.039 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-13T02:02:37.039 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T02:02:37.039 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2026-02-13T02:02:37.202 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:313: corrupt_and_repair_erasure_coded: local primary=3 2026-02-13T02:02:37.204 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2026-02-13T02:02:37.204 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: sed -e s/3// 2026-02-13T02:02:37.204 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2026-02-13T02:02:37.204 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-13T02:02:37.205 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2026-02-13T02:02:37.205 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-13T02:02:37.370 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-13T02:02:37.370 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:02:37.370 INFO:tasks.workunit.client.0.trial112.stderr:2 2026-02-13T02:02:37.370 INFO:tasks.workunit.client.0.trial112.stderr:0' 2026-02-13T02:02:37.370 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 0 2026-02-13T02:02:37.372 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: osds=('1' '2' '0') 2026-02-13T02:02:37.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: local -a osds 2026-02-13T02:02:37.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:315: corrupt_and_repair_erasure_coded: local not_primary_first=1 2026-02-13T02:02:37.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:316: corrupt_and_repair_erasure_coded: local not_primary_second=2 2026-02-13T02:02:37.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:319: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2026-02-13T02:02:37.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:286: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2026-02-13T02:02:37.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: corrupt_and_repair_one: local poolname=ecpool 2026-02-13T02:02:37.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:288: corrupt_and_repair_one: local osd=3 2026-02-13T02:02:37.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:293: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-13T02:02:37.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:02:37.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:02:37.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-13T02:02:37.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:02:37.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2026-02-13T02:02:37.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:02:37.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:02:37.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-13T02:02:37.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:02:37.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-13T02:02:37.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:02:37.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:02:37.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:02:37.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:02:37.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:02:37.681 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:02:37.681 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2026-02-13T02:02:37.681 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:02:37.681 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:02:37.681 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-13T02:02:37.681 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:02:37.681 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:02:37.681 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2026-02-13T02:02:37.871 INFO:tasks.workunit.client.0.trial112.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2026-02-13T02:02:37.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-13T02:02:37.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:02:37.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:02:37.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-13T02:02:37.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:02:37.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:02:37.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:02:37.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:02:37.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:02:37.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:02:37.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-13T02:02:37.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-13T02:02:37.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:02:37.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:02:37.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:02:37.983 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:02:37.983 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:02:37.983 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:02:37.983 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:02:37.983 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:02:37.984 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:02:37.984 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:02:37.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:02:37.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:02:37.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:02:37.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:02:37.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:02:37.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:02:37.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:02:37.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:02:37.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:02:37.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:02:37.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-13T02:02:37.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-13T02:02:37.986 INFO:tasks.workunit.client.0.trial112.stderr:start osd.3 2026-02-13T02:02:37.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:02:37.986 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-13T02:02:37.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-13T02:02:37.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:02:37.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:02:37.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:02:38.000 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:37.993+0000 7f21ede62a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:02:38.002 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:37.993+0000 7f21ede62a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:02:38.002 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:37.993+0000 7f21ede62a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:02:38.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-13T02:02:38.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:02:38.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-13T02:02:38.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:02:38.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:02:38.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:02:38.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:02:38.149 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:02:38.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:02:38.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:02:38.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:02:38.400 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:38.393+0000 7f21ede62a40 -1 Falling back to public interface 2026-02-13T02:02:38.976 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:38.969+0000 7f21ede62a40 -1 osd.3 34 log_to_monitors true 2026-02-13T02:02:39.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:02:39.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:02:39.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:02:39.313 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:02:39.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:02:39.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:02:39.472 INFO:tasks.workunit.client.0.trial112.stderr:osd.3 up in weight 1 up_from 38 up_thru 38 down_at 35 last_clean_interval [18,34) [v2:127.0.0.1:6826/677584596,v1:127.0.0.1:6827/677584596] [v2:127.0.0.1:6828/677584596,v1:127.0.0.1:6829/677584596] exists,up 61ebaa55-733a-4e41-b1bf-de710ed831df 2026-02-13T02:02:39.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:02:39.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:02:39.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:02:39.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:02:39.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:02:39.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:02:39.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:02:39.473 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:02:39.474 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:02:39.474 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:02:39.474 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:02:39.474 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:02:39.474 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:02:39.581 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:02:39.581 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:02:39.581 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:02:39.581 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:02:39.581 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:02:39.582 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:02:39.747 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:02:39.748 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:02:39.748 INFO:tasks.workunit.client.0.trial112.stderr:2 2026-02-13T02:02:39.748 INFO:tasks.workunit.client.0.trial112.stderr:3' 2026-02-13T02:02:39.748 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:02:39.748 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:02:39.748 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:02:39.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836489 2026-02-13T02:02:39.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836489 2026-02-13T02:02:39.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489' 2026-02-13T02:02:39.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:02:39.821 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:02:39.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705672 2026-02-13T02:02:39.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705672 2026-02-13T02:02:39.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-38654705672' 2026-02-13T02:02:39.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:02:39.895 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:02:39.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542151 2026-02-13T02:02:39.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542151 2026-02-13T02:02:39.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-38654705672 2-60129542151' 2026-02-13T02:02:39.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:02:39.969 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:02:40.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=163208757250 2026-02-13T02:02:40.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 163208757250 2026-02-13T02:02:40.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-38654705672 2-60129542151 3-163208757250' 2026-02-13T02:02:40.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:02:40.042 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836489 2026-02-13T02:02:40.042 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:02:40.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:02:40.044 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836489 2026-02-13T02:02:40.044 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:02:40.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836489 2026-02-13T02:02:40.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2026-02-13T02:02:40.045 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 21474836489 2026-02-13T02:02:40.046 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:02:40.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836489 2026-02-13T02:02:40.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:02:41.217 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:02:41.218 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:02:41.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836489 2026-02-13T02:02:41.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:02:41.379 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705672 2026-02-13T02:02:41.379 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:02:41.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:02:41.380 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705672 2026-02-13T02:02:41.381 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:02:41.382 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705672 2026-02-13T02:02:41.383 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705672' 2026-02-13T02:02:41.383 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 38654705672 2026-02-13T02:02:41.383 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:02:41.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705672 -lt 38654705672 2026-02-13T02:02:41.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:02:41.554 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542151 2026-02-13T02:02:41.554 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:02:41.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:02:41.556 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542151 2026-02-13T02:02:41.556 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:02:41.558 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542151 2026-02-13T02:02:41.558 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542151' 2026-02-13T02:02:41.558 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 60129542151 2026-02-13T02:02:41.558 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:02:41.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542151 -lt 60129542151 2026-02-13T02:02:41.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:02:41.732 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-163208757250 2026-02-13T02:02:41.732 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:02:41.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:02:41.734 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-163208757250 2026-02-13T02:02:41.734 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:02:41.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=163208757250 2026-02-13T02:02:41.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 163208757250' 2026-02-13T02:02:41.736 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.3 seq 163208757250 2026-02-13T02:02:41.736 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:02:41.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 163208757250 -lt 163208757250 2026-02-13T02:02:41.903 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:02:41.904 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:02:41.904 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:02:42.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:02:42.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:02:42.098 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:02:42.098 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:02:42.098 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:02:42.098 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:02:42.098 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:02:42.098 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:02:42.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:02:42.260 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:02:42.261 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:02:42.261 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:02:42.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:02:42.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:02:42.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:02:42.463 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: get_pg ecpool SOMETHING 2026-02-13T02:02:42.463 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-13T02:02:42.463 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-13T02:02:42.463 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-13T02:02:42.463 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T02:02:42.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: local pg=2.0 2026-02-13T02:02:42.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: corrupt_and_repair_one: repair 2.0 2026-02-13T02:02:42.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-13T02:02:42.628 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-13T02:02:42.629 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:02:42.629 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:02:42.629 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:02:42.629 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:02:42.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-13T02:02:31.435682+0000 2026-02-13T02:02:42.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-13T02:02:42.944 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-13T02:02:42.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-13T02:02:31.435682+0000 2026-02-13T02:02:42.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-13T02:02:42.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:02:31.435682+0000 2026-02-13T02:02:42.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:02:42.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:02:42.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:02:42.956 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:02:42.956 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:02:42.956 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:02:42.956 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:02:42.956 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:02:43.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:02:31.435682+0000 '>' 2026-02-13T02:02:31.435682+0000 2026-02-13T02:02:43.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:02:44.121 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:02:44.121 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:02:44.122 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:02:44.122 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:02:44.122 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:02:44.122 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:02:44.122 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:02:44.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:02:31.435682+0000 '>' 2026-02-13T02:02:31.435682+0000 2026-02-13T02:02:44.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:02:45.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:02:45.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:02:45.285 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:02:45.285 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:02:45.285 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:02:45.286 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:02:45.286 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:02:45.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:02:43.007466+0000 '>' 2026-02-13T02:02:31.435682+0000 2026-02-13T02:02:45.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:02:45.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-13T02:02:45.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:02:45.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:02:45.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-13T02:02:45.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:02:45.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-13T02:02:45.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:02:45.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:02:45.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-13T02:02:45.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:02:45.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-13T02:02:45.452 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:02:45.452 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:02:45.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:02:45.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:02:45.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:02:45.760 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:02:45.760 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-13T02:02:45.760 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:02:45.760 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:02:45.760 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-13T02:02:45.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:02:45.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:02:45.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2026-02-13T02:02:45.823 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::deep_scrub_2.0s0:head#, (61) No data available 2026-02-13T02:02:45.824 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2026-02-13T02:02:45.824 INFO:tasks.workunit.client.0.trial112.stdout:_ 2026-02-13T02:02:45.824 INFO:tasks.workunit.client.0.trial112.stdout:hinfo_key 2026-02-13T02:02:45.824 INFO:tasks.workunit.client.0.trial112.stdout:snapset 2026-02-13T02:02:45.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-13T02:02:45.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:02:45.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:02:45.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-13T02:02:45.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:02:45.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:02:45.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:02:45.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:02:45.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:02:45.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:02:45.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-13T02:02:45.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-13T02:02:45.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:02:45.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:02:45.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:02:45.935 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:02:45.935 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:02:45.935 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:02:45.935 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:02:45.935 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:02:45.935 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:02:45.935 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:02:45.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:02:45.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:02:45.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:02:45.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:02:45.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:02:45.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:02:45.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:02:45.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:02:45.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:02:45.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:02:45.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-13T02:02:45.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-13T02:02:45.938 INFO:tasks.workunit.client.0.trial112.stderr:start osd.3 2026-02-13T02:02:45.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:02:45.938 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-13T02:02:45.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-13T02:02:45.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:02:45.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:02:45.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:02:45.953 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:45.945+0000 7f702d39ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:02:45.954 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:45.945+0000 7f702d39ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:02:45.955 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:45.949+0000 7f702d39ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:02:46.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-13T02:02:46.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:02:46.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-13T02:02:46.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:02:46.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:02:46.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:02:46.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:02:46.102 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:02:46.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:02:46.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:02:46.266 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:02:46.301 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:46.293+0000 7f702d39ca40 -1 Falling back to public interface 2026-02-13T02:02:46.898 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:46.889+0000 7f702d39ca40 -1 osd.3 39 log_to_monitors true 2026-02-13T02:02:47.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:02:47.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:02:47.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:02:47.269 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:02:47.269 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:02:47.269 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:02:47.423 INFO:tasks.workunit.client.0.trial112.stderr:osd.3 up in weight 1 up_from 43 up_thru 43 down_at 40 last_clean_interval [38,39) [v2:127.0.0.1:6826/2857344691,v1:127.0.0.1:6827/2857344691] [v2:127.0.0.1:6828/2857344691,v1:127.0.0.1:6829/2857344691] exists,up 61ebaa55-733a-4e41-b1bf-de710ed831df 2026-02-13T02:02:47.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:02:47.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:02:47.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:02:47.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:02:47.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:02:47.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:02:47.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:02:47.424 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:02:47.425 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:02:47.425 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:02:47.425 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:02:47.425 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:02:47.425 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:02:47.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:02:47.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:02:47.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:02:47.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:02:47.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:02:47.531 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:02:47.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:02:47.693 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:02:47.693 INFO:tasks.workunit.client.0.trial112.stderr:2 2026-02-13T02:02:47.693 INFO:tasks.workunit.client.0.trial112.stderr:3' 2026-02-13T02:02:47.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:02:47.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:02:47.694 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:02:47.767 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836491 2026-02-13T02:02:47.768 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836491 2026-02-13T02:02:47.768 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491' 2026-02-13T02:02:47.768 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:02:47.768 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:02:47.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705675 2026-02-13T02:02:47.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705675 2026-02-13T02:02:47.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705675' 2026-02-13T02:02:47.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:02:47.842 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:02:47.914 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542154 2026-02-13T02:02:47.914 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542154 2026-02-13T02:02:47.914 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705675 2-60129542154' 2026-02-13T02:02:47.914 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:02:47.915 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:02:47.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=184683593730 2026-02-13T02:02:47.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 184683593730 2026-02-13T02:02:47.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705675 2-60129542154 3-184683593730' 2026-02-13T02:02:47.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:02:47.986 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836491 2026-02-13T02:02:47.987 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:02:47.988 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:02:47.988 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836491 2026-02-13T02:02:47.989 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:02:47.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836491 2026-02-13T02:02:47.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2026-02-13T02:02:47.990 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 21474836491 2026-02-13T02:02:47.990 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:02:48.160 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836490 -lt 21474836491 2026-02-13T02:02:48.160 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:02:49.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:02:49.162 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:02:49.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836491 2026-02-13T02:02:49.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:02:49.324 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705675 2026-02-13T02:02:49.324 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:02:49.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:02:49.326 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705675 2026-02-13T02:02:49.326 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:02:49.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705675 2026-02-13T02:02:49.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705675' 2026-02-13T02:02:49.328 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 38654705675 2026-02-13T02:02:49.328 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:02:49.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705675 -lt 38654705675 2026-02-13T02:02:49.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:02:49.497 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542154 2026-02-13T02:02:49.497 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:02:49.498 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:02:49.499 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542154 2026-02-13T02:02:49.499 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:02:49.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542154 2026-02-13T02:02:49.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542154' 2026-02-13T02:02:49.501 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 60129542154 2026-02-13T02:02:49.502 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:02:49.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542154 -lt 60129542154 2026-02-13T02:02:49.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:02:49.671 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-184683593730 2026-02-13T02:02:49.671 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:02:49.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:02:49.673 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-184683593730 2026-02-13T02:02:49.673 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:02:49.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=184683593730 2026-02-13T02:02:49.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 184683593730' 2026-02-13T02:02:49.675 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.3 seq 184683593730 2026-02-13T02:02:49.675 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:02:49.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 184683593730 -lt 184683593730 2026-02-13T02:02:49.845 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:02:49.846 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:02:49.846 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:02:50.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:02:50.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:02:50.038 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:02:50.039 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:02:50.039 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:02:50.039 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:02:50.039 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:02:50.039 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:02:50.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:02:50.198 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:02:50.199 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:02:50.199 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:02:50.395 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:02:50.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:02:50.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:02:50.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-13T02:02:50.414 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-13T02:02:50.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:321: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2026-02-13T02:02:50.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:286: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2026-02-13T02:02:50.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: corrupt_and_repair_one: local poolname=ecpool 2026-02-13T02:02:50.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:288: corrupt_and_repair_one: local osd=1 2026-02-13T02:02:50.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:293: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2026-02-13T02:02:50.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:02:50.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:02:50.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:02:50.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:02:50.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2026-02-13T02:02:50.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:02:50.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:02:50.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:02:50.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:02:50.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:02:50.418 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:02:50.418 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:02:50.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:02:50.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:02:50.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:02:50.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:02:50.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2026-02-13T02:02:50.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:02:50.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:02:50.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:02:50.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:02:50.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:02:50.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2026-02-13T02:02:50.915 INFO:tasks.workunit.client.0.trial112.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2026-02-13T02:02:51.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-13T02:02:51.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:02:51.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:02:51.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:02:51.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:02:51.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:02:51.015 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:02:51.015 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:02:51.015 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:02:51.015 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:02:51.015 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:02:51.015 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:02:51.015 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:02:51.015 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:02:51.015 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:02:51.015 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:02:51.015 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:02:51.016 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:02:51.016 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:02:51.016 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:02:51.016 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:02:51.016 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:02:51.016 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:02:51.016 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:02:51.016 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:02:51.016 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:02:51.016 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:02:51.016 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:02:51.016 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:02:51.017 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:02:51.017 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:02:51.017 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:02:51.017 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:02:51.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:02:51.018 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:02:51.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:02:51.018 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:02:51.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:02:51.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:02:51.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:02:51.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:02:51.034 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:51.025+0000 7f7aff5b6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:02:51.035 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:51.029+0000 7f7aff5b6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:02:51.036 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:51.029+0000 7f7aff5b6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:02:51.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:02:51.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:02:51.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:02:51.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:02:51.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:02:51.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:02:51.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:02:51.173 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:02:51.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:02:51.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:02:51.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:02:51.461 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:51.453+0000 7f7aff5b6a40 -1 Falling back to public interface 2026-02-13T02:02:52.090 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:02:52.081+0000 7f7aff5b6a40 -1 osd.1 45 log_to_monitors true 2026-02-13T02:02:52.339 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:02:52.339 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:02:52.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:02:52.340 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:02:52.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:02:52.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:02:52.499 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 49 up_thru 49 down_at 46 last_clean_interval [9,45) [v2:127.0.0.1:6810/3971051540,v1:127.0.0.1:6811/3971051540] [v2:127.0.0.1:6812/3971051540,v1:127.0.0.1:6813/3971051540] exists,up 776209f2-682b-43d3-a00a-d3186f312ec7 2026-02-13T02:02:52.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:02:52.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:02:52.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:02:52.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:02:52.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:02:52.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:02:52.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:02:52.500 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:02:52.501 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:02:52.501 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:02:52.501 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:02:52.501 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:02:52.501 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:02:52.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:02:52.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:02:52.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:02:52.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:02:52.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:02:52.606 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:02:52.773 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:02:52.773 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:02:52.774 INFO:tasks.workunit.client.0.trial112.stderr:2 2026-02-13T02:02:52.774 INFO:tasks.workunit.client.0.trial112.stderr:3' 2026-02-13T02:02:52.774 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:02:52.774 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:02:52.774 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:02:52.847 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836493 2026-02-13T02:02:52.847 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836493 2026-02-13T02:02:52.847 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493' 2026-02-13T02:02:52.847 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:02:52.847 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:02:52.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=210453397506 2026-02-13T02:02:52.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 210453397506 2026-02-13T02:02:52.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-210453397506' 2026-02-13T02:02:52.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:02:52.919 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:02:52.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542156 2026-02-13T02:02:52.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542156 2026-02-13T02:02:52.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-210453397506 2-60129542156' 2026-02-13T02:02:52.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:02:52.992 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:02:53.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=184683593732 2026-02-13T02:02:53.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 184683593732 2026-02-13T02:02:53.064 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-210453397506 2-60129542156 3-184683593732' 2026-02-13T02:02:53.064 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:02:53.064 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836493 2026-02-13T02:02:53.064 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:02:53.065 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:02:53.066 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836493 2026-02-13T02:02:53.066 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:02:53.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836493 2026-02-13T02:02:53.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2026-02-13T02:02:53.068 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 21474836493 2026-02-13T02:02:53.068 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:02:53.230 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836493 -lt 21474836493 2026-02-13T02:02:53.230 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:02:53.230 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-210453397506 2026-02-13T02:02:53.231 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:02:53.232 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:02:53.233 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-210453397506 2026-02-13T02:02:53.233 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:02:53.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=210453397506 2026-02-13T02:02:53.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 210453397506' 2026-02-13T02:02:53.234 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 210453397506 2026-02-13T02:02:53.235 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:02:53.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 210453397506 2026-02-13T02:02:53.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:02:54.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:02:54.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:02:54.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 210453397506 2026-02-13T02:02:54.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:02:55.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:02:55.568 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:02:55.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 210453397506 -lt 210453397506 2026-02-13T02:02:55.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:02:55.734 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542156 2026-02-13T02:02:55.734 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:02:55.735 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:02:55.736 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542156 2026-02-13T02:02:55.736 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:02:55.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542156 2026-02-13T02:02:55.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542156' 2026-02-13T02:02:55.738 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 60129542156 2026-02-13T02:02:55.738 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:02:55.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542156 -lt 60129542156 2026-02-13T02:02:55.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:02:55.911 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-184683593732 2026-02-13T02:02:55.911 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:02:55.913 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:02:55.914 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-184683593732 2026-02-13T02:02:55.914 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:02:55.915 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=184683593732 2026-02-13T02:02:55.915 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 184683593732' 2026-02-13T02:02:55.915 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.3 seq 184683593732 2026-02-13T02:02:55.916 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:02:56.079 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 184683593732 -lt 184683593732 2026-02-13T02:02:56.080 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:02:56.081 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:02:56.081 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:02:56.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:02:56.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:02:56.280 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:02:56.280 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:02:56.280 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:02:56.280 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:02:56.280 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:02:56.280 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:02:56.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:02:56.440 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:02:56.441 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:02:56.441 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:02:56.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:02:56.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:02:56.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:02:56.637 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: get_pg ecpool SOMETHING 2026-02-13T02:02:56.637 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-13T02:02:56.637 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-13T02:02:56.637 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-13T02:02:56.637 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T02:02:56.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: local pg=2.0 2026-02-13T02:02:56.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: corrupt_and_repair_one: repair 2.0 2026-02-13T02:02:56.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-13T02:02:56.803 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-13T02:02:56.804 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:02:56.804 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:02:56.804 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:02:56.804 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:02:56.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-13T02:02:43.007466+0000 2026-02-13T02:02:56.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-13T02:02:57.114 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-13T02:02:57.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-13T02:02:43.007466+0000 2026-02-13T02:02:57.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-13T02:02:57.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:02:43.007466+0000 2026-02-13T02:02:57.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:02:57.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:02:57.125 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:02:57.125 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:02:57.125 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:02:57.125 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:02:57.125 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:02:57.125 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:02:57.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:02:43.007466+0000 '>' 2026-02-13T02:02:43.007466+0000 2026-02-13T02:02:57.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:02:58.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:02:58.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:02:58.288 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:02:58.289 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:02:58.289 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:02:58.289 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:02:58.289 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:02:58.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:02:43.007466+0000 '>' 2026-02-13T02:02:43.007466+0000 2026-02-13T02:02:58.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:02:59.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:02:59.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:02:59.452 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:02:59.452 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:02:59.452 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:02:59.453 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:02:59.453 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:02:59.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:02:43.007466+0000 '>' 2026-02-13T02:02:43.007466+0000 2026-02-13T02:02:59.615 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:03:00.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:03:00.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:03:00.617 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:03:00.617 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:03:00.617 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:03:00.617 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:03:00.617 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:03:00.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:02:43.007466+0000 '>' 2026-02-13T02:02:43.007466+0000 2026-02-13T02:03:00.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:03:01.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:03:01.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:03:01.779 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:03:01.779 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:03:01.779 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:03:01.779 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:03:01.779 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:03:01.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:02:43.007466+0000 '>' 2026-02-13T02:02:43.007466+0000 2026-02-13T02:03:01.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:03:02.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:03:02.944 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:03:02.945 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:03:02.945 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:03:02.945 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:03:02.945 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:03:02.945 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:03:03.108 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:02:57.825878+0000 '>' 2026-02-13T02:02:43.007466+0000 2026-02-13T02:03:03.108 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:03:03.108 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-13T02:03:03.108 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:03:03.108 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:03:03.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-13T02:03:03.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:03:03.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-13T02:03:03.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:03:03.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:03:03.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-13T02:03:03.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:03:03.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-13T02:03:03.109 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:03:03.109 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:03:03.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:03:03.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:03:03.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:03:03.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:03:03.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-13T02:03:03.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:03:03.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:03:03.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-13T02:03:03.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:03:03.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:03:03.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2026-02-13T02:03:03.478 INFO:tasks.workunit.client.0.trial112.stdout:_ 2026-02-13T02:03:03.478 INFO:tasks.workunit.client.0.trial112.stdout:hinfo_key 2026-02-13T02:03:03.478 INFO:tasks.workunit.client.0.trial112.stdout:snapset 2026-02-13T02:03:03.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-13T02:03:03.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:03:03.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:03:03.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-13T02:03:03.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:03:03.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:03:03.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:03:03.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:03:03.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:03:03.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:03:03.584 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-13T02:03:03.584 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-13T02:03:03.584 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:03:03.584 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:03:03.584 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:03:03.584 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:03:03.584 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:03:03.584 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:03:03.584 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:03:03.584 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:03:03.584 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:03:03.584 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:03:03.585 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:03:03.585 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:03:03.585 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:03:03.585 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:03:03.585 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:03:03.585 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:03:03.585 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:03:03.585 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:03:03.585 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:03:03.585 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:03:03.585 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-13T02:03:03.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-13T02:03:03.586 INFO:tasks.workunit.client.0.trial112.stderr:start osd.3 2026-02-13T02:03:03.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:03:03.586 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-13T02:03:03.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-13T02:03:03.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:03:03.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:03:03.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:03:03.604 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:03.597+0000 7f16e4a3fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:03.605 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:03.597+0000 7f16e4a3fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:03.606 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:03.597+0000 7f16e4a3fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:03.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-13T02:03:03.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:03:03.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-13T02:03:03.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:03:03.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:03:03.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:03:03.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:03:03.744 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:03:03.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:03:03.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:03:03.906 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:03:03.981 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:03.972+0000 7f16e4a3fa40 -1 Falling back to public interface 2026-02-13T02:03:04.545 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:04.536+0000 7f16e4a3fa40 -1 osd.3 51 log_to_monitors true 2026-02-13T02:03:04.908 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:03:04.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:03:04.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:03:04.909 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:03:04.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:03:04.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:03:05.068 INFO:tasks.workunit.client.0.trial112.stderr:osd.3 up in weight 1 up_from 54 up_thru 54 down_at 51 last_clean_interval [43,51) [v2:127.0.0.1:6826/2396769793,v1:127.0.0.1:6827/2396769793] [v2:127.0.0.1:6828/2396769793,v1:127.0.0.1:6829/2396769793] exists,up 61ebaa55-733a-4e41-b1bf-de710ed831df 2026-02-13T02:03:05.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:03:05.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:03:05.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:03:05.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:03:05.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:03:05.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:03:05.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:03:05.070 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:03:05.070 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:03:05.070 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:03:05.070 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:03:05.070 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:03:05.070 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:03:05.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:03:05.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:03:05.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:03:05.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:03:05.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:03:05.186 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:03:05.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:03:05.351 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:03:05.352 INFO:tasks.workunit.client.0.trial112.stderr:2 2026-02-13T02:03:05.352 INFO:tasks.workunit.client.0.trial112.stderr:3' 2026-02-13T02:03:05.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:03:05.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:05.352 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:03:05.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836497 2026-02-13T02:03:05.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836497 2026-02-13T02:03:05.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497' 2026-02-13T02:03:05.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:05.426 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:03:05.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=210453397509 2026-02-13T02:03:05.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 210453397509 2026-02-13T02:03:05.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-210453397509' 2026-02-13T02:03:05.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:05.501 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:03:05.576 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542159 2026-02-13T02:03:05.576 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542159 2026-02-13T02:03:05.576 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-210453397509 2-60129542159' 2026-02-13T02:03:05.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:05.577 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:03:05.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=231928233986 2026-02-13T02:03:05.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 231928233986 2026-02-13T02:03:05.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-210453397509 2-60129542159 3-231928233986' 2026-02-13T02:03:05.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:05.650 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836497 2026-02-13T02:03:05.650 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:05.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:03:05.653 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836497 2026-02-13T02:03:05.653 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:05.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836497 2026-02-13T02:03:05.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2026-02-13T02:03:05.654 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 21474836497 2026-02-13T02:03:05.655 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:03:05.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836496 -lt 21474836497 2026-02-13T02:03:05.827 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:03:06.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:03:06.828 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:03:06.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836497 -lt 21474836497 2026-02-13T02:03:06.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:06.994 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-210453397509 2026-02-13T02:03:06.995 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:06.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:03:06.997 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-210453397509 2026-02-13T02:03:06.997 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:06.998 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=210453397509 2026-02-13T02:03:06.998 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 210453397509' 2026-02-13T02:03:06.999 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 210453397509 2026-02-13T02:03:06.999 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:03:07.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 210453397509 -lt 210453397509 2026-02-13T02:03:07.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:07.169 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542159 2026-02-13T02:03:07.169 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:07.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:03:07.171 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542159 2026-02-13T02:03:07.172 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:07.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542159 2026-02-13T02:03:07.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542159' 2026-02-13T02:03:07.173 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 60129542159 2026-02-13T02:03:07.174 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:03:07.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542160 -lt 60129542159 2026-02-13T02:03:07.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:07.342 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-231928233986 2026-02-13T02:03:07.343 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:07.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:03:07.344 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-231928233986 2026-02-13T02:03:07.345 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:07.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=231928233986 2026-02-13T02:03:07.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 231928233986' 2026-02-13T02:03:07.346 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.3 seq 231928233986 2026-02-13T02:03:07.347 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:03:07.515 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 231928233986 -lt 231928233986 2026-02-13T02:03:07.515 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:03:07.516 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:03:07.516 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:03:07.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:03:07.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:03:07.709 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:03:07.709 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:03:07.709 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:03:07.709 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:03:07.709 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:03:07.710 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:03:07.871 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:03:07.871 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:03:07.872 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:03:07.872 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:03:08.072 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:03:08.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:03:08.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:03:08.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-13T02:03:08.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-13T02:03:08.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:322: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2026-02-13T02:03:08.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:245: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2026-02-13T02:03:08.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:246: corrupt_and_repair_two: local poolname=ecpool 2026-02-13T02:03:08.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:247: corrupt_and_repair_two: local first=1 2026-02-13T02:03:08.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_two: local second=2 2026-02-13T02:03:08.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_two: pids= 2026-02-13T02:03:08.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2026-02-13T02:03:08.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-13T02:03:08.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-13T02:03:08.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 127052"' 2026-02-13T02:03:08.095 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 127052' 2026-02-13T02:03:08.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2026-02-13T02:03:08.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-13T02:03:08.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-13T02:03:08.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 127053"' 2026-02-13T02:03:08.095 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 127053' 2026-02-13T02:03:08.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_two: wait_background pids 2026-02-13T02:03:08.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 127052 127053' 2026-02-13T02:03:08.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-13T02:03:08.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-13T02:03:08.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 127052 2026-02-13T02:03:08.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2026-02-13T02:03:08.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/127055: /' 2026-02-13T02:03:08.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2026-02-13T02:03:08.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/127057: /' 2026-02-13T02:03:08.688 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:03:08.688 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:03:08.688 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:03:08.688 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:03:08.688 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2026-02-13T02:03:08.689 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:03:08.689 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:03:08.689 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:03:08.689 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:03:08.689 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:03:08.689 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:03:08.689 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:03:08.689 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:03:08.689 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:03:08.689 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:03:08.689 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:03:08.689 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2026-02-13T02:03:08.690 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:03:08.690 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:03:08.690 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:03:08.690 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:03:08.690 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:03:08.690 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2026-02-13T02:03:08.690 INFO:tasks.workunit.client.0.trial112.stderr:127055: remove 1#2:eb822e21:::SOMETHING:head# 2026-02-13T02:03:08.690 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-13T02:03:08.690 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:03:08.690 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:03:08.690 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:03:08.690 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:03:08.691 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:03:08.691 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:03:08.691 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:03:08.691 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: 127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:03:08.691 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:03:08.692 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-13T02:03:08.692 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:03:08.692 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2026-02-13T02:03:08.692 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:03:08.692 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:03:08.692 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-13T02:03:08.692 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:03:08.692 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2026-02-13T02:03:08.692 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:03:08.692 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:03:08.692 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:03:08.692 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:03:08.693 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:03:08.693 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:03:08.693 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2026-02-13T02:03:08.693 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:03:08.693 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:03:08.693 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-13T02:03:08.693 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:03:08.693 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2026-02-13T02:03:08.693 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2026-02-13T02:03:08.693 INFO:tasks.workunit.client.0.trial112.stderr:127057: remove 2#2:eb822e21:::SOMETHING:head# 2026-02-13T02:03:08.693 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2026-02-13T02:03:08.693 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:03:08.694 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:03:08.694 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-13T02:03:08.694 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:03:08.694 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2026-02-13T02:03:08.694 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:03:08.694 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:03:08.695 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:03:08.695 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:03:08.695 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:03:08.695 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:03:08.695 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:03:08.695 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:03:08.695 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:03:08.695 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:03:08.695 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:03:08.696 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:03:08.696 INFO:tasks.workunit.client.0.trial112.stderr:127055: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:03:08.696 INFO:tasks.workunit.client.0.trial112.stderr:127055: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:03:08.696 INFO:tasks.workunit.client.0.trial112.stderr:127055: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:03:08.696 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:03:08.696 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:03:08.696 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:03:08.696 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:03:08.696 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:03:08.696 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:03:08.696 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:03:08.696 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:03:08.697 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:03:08.697 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:03:08.697 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:03:08.697 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:03:08.697 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:03:08.697 INFO:tasks.workunit.client.0.trial112.stderr:127055: start osd.1 2026-02-13T02:03:08.697 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:03:08.697 INFO:tasks.workunit.client.0.trial112.stderr:1activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:03:08.697 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:03:08.697 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-13T02:03:08.697 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-13T02:03:08.698 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:03:08.698 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:03:08.698 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:03:08.698 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:03:08.698 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:03:08.698 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:03:08.698 INFO:tasks.workunit.client.0.trial112.stderr:127057: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:03:08.698 INFO:tasks.workunit.client.0.trial112.stderr:127057: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:03:08.698 INFO:tasks.workunit.client.0.trial112.stderr:127057: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:03:08.698 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:03:08.698 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:03:08.698 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:03:08.699 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:03:08.699 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:03:08.699 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:03:08.699 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:03:08.699 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:03:08.699 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:03:08.699 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:03:08.699 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:03:08.699 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2026-02-13T02:03:08.699 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-13T02:03:08.699 INFO:tasks.workunit.client.0.trial112.stderr:127057: start osd.2 2026-02-13T02:03:08.699 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:03:10.176 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2026-02-13T02:03:10.176 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-13T02:03:10.176 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:03:10.176 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:03:10.176 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:03:10.176 INFO:tasks.workunit.client.0.trial112.stderr:127057: 2026-02-13T02:03:08.696+0000 7f162e9fea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:10.176 INFO:tasks.workunit.client.0.trial112.stderr:127057: 2026-02-13T02:03:08.700+0000 7f162e9fea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:10.176 INFO:tasks.workunit.client.0.trial112.stderr:127057: 2026-02-13T02:03:08.700+0000 7f162e9fea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:10.177 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-13T02:03:10.177 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:03:10.177 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T02:03:10.177 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:03:10.177 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:03:10.177 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:03:10.177 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:03:10.177 INFO:tasks.workunit.client.0.trial112.stderr:127057: 0 2026-02-13T02:03:10.177 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:03:10.177 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:03:10.177 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:03:10.177 INFO:tasks.workunit.client.0.trial112.stderr:127057: 2026-02-13T02:03:09.224+0000 7f162e9fea40 -1 Falling back to public interface 2026-02-13T02:03:10.178 INFO:tasks.workunit.client.0.trial112.stderr:127057: 2026-02-13T02:03:09.820+0000 7f162e9fea40 -1 osd.2 56 log_to_monitors true 2026-02-13T02:03:10.178 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:03:10.178 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:03:10.178 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:03:10.178 INFO:tasks.workunit.client.0.trial112.stderr:127057: 1 2026-02-13T02:03:10.178 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:03:10.178 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:03:10.178 INFO:tasks.workunit.client.0.trial112.stderr:127057: osd.2 up in weight 1 up_from 60 up_thru 46 down_at 57 last_clean_interval [14,56) [v2:127.0.0.1:6810/2549712664,v1:127.0.0.1:6811/2549712664] [v2:127.0.0.1:6812/2549712664,v1:127.0.0.1:6813/2549712664] exists,up 6d9c196a-4cd8-413e-bba5-8ecb673833b9 2026-02-13T02:03:10.178 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:03:10.178 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:03:10.178 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:03:10.178 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:03:10.179 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:03:10.179 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:03:10.179 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:03:10.179 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:03:10.179 INFO:tasks.workunit.client.0.trial112.stderr:127057: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:03:10.184 INFO:tasks.workunit.client.0.trial112.stderr:127057: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: g27055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:03:10.184 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:03:10.184 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:03:10.184 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:03:10.184 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:03:10.184 INFO:tasks.workunit.client.0.trial112.stderr:127055: 2026-02-13T02:03:08.696+0000 7fa3d51c7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:10.185 INFO:tasks.workunit.client.0.trial112.stderr:127055: 2026-02-13T02:03:08.700+0000 7fa3d51c7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:10.185 INFO:tasks.workunit.client.0.trial112.stderr:127055: 2026-02-13T02:03:08.700+0000 7fa3d51c7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:10.185 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:03:10.185 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:03:10.185 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:03:10.185 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:03:10.185 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:03:10.185 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:03:10.185 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:03:10.185 INFO:tasks.workunit.client.0.trial112.stderr:127055: 0 2026-02-13T02:03:10.185 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:03:10.185 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:03:10.186 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:03:10.186 INFO:tasks.workunit.client.0.trial112.stderr:127055: 2026-02-13T02:03:09.232+0000 7fa3d51c7a40 -1 Falling back to public interface 2026-02-13T02:03:10.186 INFO:tasks.workunit.client.0.trial112.stderr:127055: 2026-02-13T02:03:09.868+0000 7fa3d51c7a40 -1 osd.1 56 log_to_monitors true 2026-02-13T02:03:10.186 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:03:10.186 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:03:10.186 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:03:10.186 INFO:tasks.workunit.client.0.trial112.stderr:127055: 1 2026-02-13T02:03:10.186 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:03:10.186 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:03:10.186 INFO:tasks.workunit.client.0.trial112.stderr:127055: osd.1 up in weight 1 up_from 60 up_thru 60 down_at 57 last_clean_interval [49,56) [v2:127.0.0.1:6818/733506697,v1:127.0.0.1:6819/733506697] [v2:127.0.0.1:6820/733506697,v1:127.0.0.1:6821/733506697] exists,up 776209f2-682b-43d3-a00a-d3186f312ec7 2026-02-13T02:03:10.187 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:03:10.187 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:03:10.187 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:03:10.187 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:03:10.187 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:03:10.187 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:03:10.187 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:03:10.187 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:03:10.187 INFO:tasks.workunit.client.0.trial112.stderr:127055: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:03:10.729 INFO:tasks.workunit.client.0.trial112.stderr:127055: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_tet_timeout_delays: echo true 2026-02-13T02:03:10.729 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:03:10.729 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:03:10.729 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:03:10.730 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:03:10.730 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:03:10.730 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:03:10.730 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:03:10.730 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:03:10.730 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:03:10.730 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:03:10.730 INFO:tasks.workunit.client.0.trial112.stderr:127057: 1 2026-02-13T02:03:10.730 INFO:tasks.workunit.client.0.trial112.stderr:127057: 2 2026-02-13T02:03:10.730 INFO:tasks.workunit.client.0.trial112.stderr:127057: 3' 2026-02-13T02:03:10.730 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:03:10.730 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:10.731 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:03:10.731 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836499 2026-02-13T02:03:10.731 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836499 2026-02-13T02:03:10.731 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499' 2026-02-13T02:03:10.731 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:10.731 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:03:10.731 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=257698037762 2026-02-13T02:03:10.731 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 257698037762 2026-02-13T02:03:10.731 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-257698037762' 2026-02-13T02:03:10.731 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:10.731 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:03:10.731 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=257698037762 2026-02-13T02:03:10.732 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 257698037762 2026-02-13T02:03:10.732 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-257698037762 2-257698037762' 2026-02-13T02:03:10.732 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:10.732 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:03:10.732 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=231928233988 2026-02-13T02:03:10.732 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 231928233988 2026-02-13T02:03:10.732 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-257698037762 2-257698037762 3-231928233988' 2026-02-13T02:03:10.732 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:10.732 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836499 2026-02-13T02:03:10.754 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/imeout_delays: echo true 2026-02-13T02:03:10.755 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:03:10.755 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:03:10.755 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:03:10.755 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:03:10.755 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:03:10.755 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:03:10.755 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:03:10.755 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:03:10.755 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:03:10.755 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:03:10.755 INFO:tasks.workunit.client.0.trial112.stderr:127055: 1 2026-02-13T02:03:10.756 INFO:tasks.workunit.client.0.trial112.stderr:127055: 2 2026-02-13T02:03:10.756 INFO:tasks.workunit.client.0.trial112.stderr:127055: 3' 2026-02-13T02:03:10.756 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:03:10.756 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:10.756 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:03:10.756 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836500 2026-02-13T02:03:10.756 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836500 2026-02-13T02:03:10.756 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836500' 2026-02-13T02:03:10.756 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:10.756 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:03:10.757 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=257698037763 2026-02-13T02:03:10.757 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 257698037763 2026-02-13T02:03:10.757 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836500 1-257698037763' 2026-02-13T02:03:10.757 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:10.757 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:03:10.757 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=257698037763 2026-02-13T02:03:10.757 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 257698037763 2026-02-13T02:03:10.757 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836500 1-257698037763 2-257698037763' 2026-02-13T02:03:10.757 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:10.757 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:03:10.757 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=231928233989 2026-02-13T02:03:10.757 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 231928233989 2026-02-13T02:03:10.758 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836500 1-257698037763 2-257698037763 3-231928233989' 2026-02-13T02:03:10.758 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:10.758 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836500 2026-02-13T02:03:12.388 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubunubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:12.388 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:03:12.388 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836499 2026-02-13T02:03:12.389 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:12.389 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836499 2026-02-13T02:03:12.389 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2026-02-13T02:03:12.389 INFO:tasks.workunit.client.0.trial112.stderr:127057: waiting osd.0 seq 21474836499 2026-02-13T02:03:12.389 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:03:12.389 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836497 -lt 21474836499 2026-02-13T02:03:12.389 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:03:12.389 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:03:12.389 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:03:12.389 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836500 -lt 21474836499 2026-02-13T02:03:12.389 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:12.389 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-257698037762 2026-02-13T02:03:12.390 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:12.390 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:03:12.390 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-257698037762 2026-02-13T02:03:12.390 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:12.390 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=257698037762 2026-02-13T02:03:12.390 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 257698037762' 2026-02-13T02:03:12.390 INFO:tasks.workunit.client.0.trial112.stderr:127057: waiting osd.1 seq 257698037762 2026-02-13T02:03:12.390 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:03:12.390 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 257698037763 -lt 257698037762 2026-02-13T02:03:12.390 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:12.390 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-257698037762 2026-02-13T02:03:12.390 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:12.391 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:03:12.391 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-257698037762 2026-02-13T02:03:12.391 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:12.391 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=257698037762 2026-02-13T02:03:12.391 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 257698037762' 2026-02-13T02:03:12.391 INFO:tasks.workunit.client.0.trial112.stderr:127057: waiting osd.2 seq 257698037762 2026-02-13T02:03:12.391 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:03:12.391 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 257698037763 -lt 257698037762 2026-02-13T02:03:12.391 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:12.426 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.shtu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:12.427 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:03:12.427 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836500 2026-02-13T02:03:12.427 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:12.427 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836500 2026-02-13T02:03:12.427 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2026-02-13T02:03:12.427 INFO:tasks.workunit.client.0.trial112.stderr:127055: waiting osd.0 seq 21474836500 2026-02-13T02:03:12.427 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:03:12.427 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836497 -lt 21474836500 2026-02-13T02:03:12.427 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:03:12.427 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:03:12.427 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:03:12.428 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836500 -lt 21474836500 2026-02-13T02:03:12.428 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:12.428 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-257698037763 2026-02-13T02:03:12.428 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:12.428 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:03:12.428 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-257698037763 2026-02-13T02:03:12.428 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:12.428 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=257698037763 2026-02-13T02:03:12.428 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 257698037763' 2026-02-13T02:03:12.428 INFO:tasks.workunit.client.0.trial112.stderr:127055: waiting osd.1 seq 257698037763 2026-02-13T02:03:12.428 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:03:12.428 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 257698037763 -lt 257698037763 2026-02-13T02:03:12.428 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:12.429 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-257698037763 2026-02-13T02:03:12.429 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:12.429 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:03:12.429 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-257698037763 2026-02-13T02:03:12.429 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:12.429 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=257698037763 2026-02-13T02:03:12.429 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 257698037763' 2026-02-13T02:03:12.429 INFO:tasks.workunit.client.0.trial112.stderr:127055: waiting osd.2 seq 257698037763 2026-02-13T02:03:12.429 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:03:12.429 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 257698037763 -lt 257698037763 2026-02-13T02:03:12.429 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:13.109 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:226:2267: flush_pg_stats: echo 3-231928233988 2026-02-13T02:03:13.109 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:13.109 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:03:13.109 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-231928233988 2026-02-13T02:03:13.109 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:13.109 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=231928233988 2026-02-13T02:03:13.109 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 231928233988' 2026-02-13T02:03:13.110 INFO:tasks.workunit.client.0.trial112.stderr:127057: waiting osd.3 seq 231928233988 2026-02-13T02:03:13.110 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:03:13.110 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 231928233989 -lt 231928233988 2026-02-13T02:03:13.110 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:03:13.110 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:03:13.110 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:03:13.110 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:03:13.110 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:03:13.110 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:03:13.110 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:03:13.110 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:03:13.110 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:03:13.111 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:03:13.111 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:03:13.111 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:03:13.111 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:03:13.111 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:03:13.111 INFO:tasks.workunit.client.0.trial112.stderr:127057: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:03:13.111 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:03:13.111 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:03:13.111 INFO:tasks.workunit.client.0.trial112.stderr:127057: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:03:13.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-13T02:03:13.146 INFO:tasks.workunit.client.0.trial112.stderr:7: flush_pg_stats: echo 3-231928233989 2026-02-13T02:03:13.146 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:13.146 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:03:13.147 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-231928233989 2026-02-13T02:03:13.147 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:13.147 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=231928233989 2026-02-13T02:03:13.147 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 231928233989' 2026-02-13T02:03:13.147 INFO:tasks.workunit.client.0.trial112.stderr:127055: waiting osd.3 seq 231928233989 2026-02-13T02:03:13.147 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:03:13.147 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 231928233989 -lt 231928233989 2026-02-13T02:03:13.147 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:03:13.147 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:03:13.147 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:03:13.147 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:03:13.147 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:03:13.148 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:03:13.148 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:03:13.148 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:03:13.148 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:03:13.148 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:03:13.148 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:03:13.148 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:03:13.148 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:03:13.148 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:03:13.148 INFO:tasks.workunit.client.0.trial112.stderr:127055: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:03:13.148 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:03:13.148 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:03:13.148 INFO:tasks.workunit.client.0.trial112.stderr:127055: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:03:13.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-13T02:03:13.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-13T02:03:13.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 127053 2026-02-13T02:03:13.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-13T02:03:13.150 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-13T02:03:13.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-13T02:03:13.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:257: corrupt_and_repair_two: return_code=0 2026-02-13T02:03:13.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:258: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-13T02:03:13.151 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: get_pg ecpool SOMETHING 2026-02-13T02:03:13.151 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-13T02:03:13.151 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-13T02:03:13.151 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-13T02:03:13.151 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T02:03:13.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: local pg=2.0 2026-02-13T02:03:13.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:264: corrupt_and_repair_two: repair 2.0 2026-02-13T02:03:13.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-13T02:03:13.314 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-13T02:03:13.314 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:03:13.315 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:03:13.315 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:03:13.315 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:03:13.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-13T02:02:57.825878+0000 2026-02-13T02:03:13.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-13T02:03:13.627 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-13T02:03:13.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-13T02:02:57.825878+0000 2026-02-13T02:03:13.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-13T02:03:13.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:02:57.825878+0000 2026-02-13T02:03:13.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:03:13.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:03:13.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:03:13.638 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:03:13.638 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:03:13.638 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:03:13.639 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:03:13.639 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:03:13.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:02:57.825878+0000 '>' 2026-02-13T02:02:57.825878+0000 2026-02-13T02:03:13.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:03:14.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:03:14.806 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:03:14.806 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:03:14.807 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:03:14.807 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:03:14.807 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:03:14.807 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:03:14.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:03:14.427998+0000 '>' 2026-02-13T02:02:57.825878+0000 2026-02-13T02:03:14.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:03:14.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: corrupt_and_repair_two: pids= 2026-02-13T02:03:14.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-13T02:03:14.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-13T02:03:14.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-13T02:03:14.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 129789"' 2026-02-13T02:03:14.973 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 129789' 2026-02-13T02:03:14.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2026-02-13T02:03:14.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-13T02:03:14.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-13T02:03:14.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 129791"' 2026-02-13T02:03:14.973 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 129791' 2026-02-13T02:03:14.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-13T02:03:14.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:271: corrupt_and_repair_two: wait_background pids 2026-02-13T02:03:14.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 129789 129791' 2026-02-13T02:03:14.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-13T02:03:14.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-13T02:03:14.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 129789 2026-02-13T02:03:14.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/129792: /' 2026-02-13T02:03:14.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2026-02-13T02:03:14.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/129794: /' 2026-02-13T02:03:15.460 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:03:15.460 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:03:15.460 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-13T02:03:15.460 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:03:15.460 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2026-02-13T02:03:15.460 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:03:15.460 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:03:15.460 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-13T02:03:15.460 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:03:15.461 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2026-02-13T02:03:15.461 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:03:15.461 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:03:15.461 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:03:15.461 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:03:15.461 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:03:15.461 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:03:15.461 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2026-02-13T02:03:15.461 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:03:15.461 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:03:15.461 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-13T02:03:15.461 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:03:15.462 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2026-02-13T02:03:15.462 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2026-02-13T02:03:15.462 INFO:tasks.workunit.client.0.trial112.stderr:129794: _ 2026-02-13T02:03:15.462 INFO:tasks.workunit.client.0.trial112.stderr:129794: hinfo_key 2026-02-13T02:03:15.462 INFO:tasks.workunit.client.0.trial112.stderr:129794: snapset 2026-02-13T02:03:15.462 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2026-02-13T02:03:15.462 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:03:15.462 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:03:15.462 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-13T02:03:15.462 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:03:15.462 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2026-02-13T02:03:15.462 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:03:15.463 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:03:15.463 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:03:15.463 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:03:15.463 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-13T02:03:15.464 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-13T02:03:15.464 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:03:15.464 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:03:15.464 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:03:15.464 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:03:15.464 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:03:15.464 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:03:15.464 INFO:tasks.workunit.client.0.trial112.stderr:129794: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:03:15.464 INFO:tasks.workunit.client.0.trial112.stderr:129794: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:03:15.464 INFO:tasks.workunit.client.0.trial112.stderr:129794: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:03:15.464 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:03:15.464 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:03:15.465 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:03:15.465 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:03:15.465 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:03:15.465 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:03:15.465 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:03:15.465 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:03:15.465 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:03:15.465 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:03:15.465 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:03:15.465 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2026-02-13T02:03:15.465 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-13T02:03:15.465 INFO:tasks.workunit.client.0.trial112.stderr:129794: start osd.2 2026-02-13T02:03:15.476 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reco129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:03:15.476 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:03:15.476 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:03:15.476 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:03:15.476 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-13T02:03:15.476 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:03:15.476 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:03:15.477 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:03:15.477 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:03:15.477 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:03:15.477 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:03:15.477 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:03:15.477 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:03:15.477 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:03:15.477 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:03:15.477 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:03:15.477 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-13T02:03:15.477 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:03:15.477 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:03:15.478 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:03:15.478 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:03:15.478 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:03:15.478 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2026-02-13T02:03:15.478 INFO:tasks.workunit.client.0.trial112.stderr:129792: _ 2026-02-13T02:03:15.478 INFO:tasks.workunit.client.0.trial112.stderr:129792: hinfo_key 2026-02-13T02:03:15.478 INFO:tasks.workunit.client.0.trial112.stderr:129792: snapset 2026-02-13T02:03:15.478 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-13T02:03:15.478 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:03:15.478 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:03:15.478 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:03:15.478 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:03:15.479 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:03:15.479 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:03:15.479 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:03:15.480 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:03:15.480 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:03:15.480 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:03:15.480 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:03:15.480 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:03:15.480 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:03:15.480 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:03:15.480 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:03:15.480 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:03:15.480 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:03:15.481 INFO:tasks.workunit.client.0.trial112.stderr:129792: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:03:15.481 INFO:tasks.workunit.client.0.trial112.stderr:129792: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:03:15.481 INFO:tasks.workunit.client.0.trial112.stderr:129792: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:03:15.481 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:03:15.481 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:03:15.481 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:03:15.481 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:03:15.481 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:03:15.481 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:03:15.481 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:03:15.481 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:03:15.481 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:03:15.482 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:03:15.482 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:03:15.482 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:03:15.482 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:03:15.482 INFO:tasks.workunit.client.0.trial112.stderr:129792: start osd.1 2026-02-13T02:03:16.954 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:03:16.954 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2026-02-13T02:03:16.954 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-13T02:03:16.954 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:03:16.954 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:03:16.955 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:03:16.955 INFO:tasks.workunit.client.0.trial112.stderr:129794: 2026-02-13T02:03:15.476+0000 7f2b61c51a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:16.955 INFO:tasks.workunit.client.0.trial112.stderr:129794: 2026-02-13T02:03:15.476+0000 7f2b61c51a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:16.955 INFO:tasks.workunit.client.0.trial112.stderr:129794: 2026-02-13T02:03:15.480+0000 7f2b61c51a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:16.955 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-13T02:03:16.955 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:03:16.955 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T02:03:16.955 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:03:16.955 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:03:16.955 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:03:16.955 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:03:16.955 INFO:tasks.workunit.client.0.trial112.stderr:129794: 0 2026-02-13T02:03:16.956 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:03:16.956 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:03:16.956 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:03:16.956 INFO:tasks.workunit.client.0.trial112.stderr:129794: 2026-02-13T02:03:15.952+0000 7f2b61c51a40 -1 Falling back to public interface 2026-02-13T02:03:16.956 INFO:tasks.workunit.client.0.trial112.stderr:129794: 2026-02-13T02:03:16.624+0000 7f2b61c51a40 -1 osd.2 63 log_to_monitors true 2026-02-13T02:03:16.956 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:03:16.956 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:03:16.956 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:03:16.956 INFO:tasks.workunit.client.0.trial112.stderr:129794: 1 2026-02-13T02:03:16.956 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:03:16.956 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:03:16.956 INFO:tasks.workunit.client.0.trial112.stderr:129794: osd.2 up in weight 1 up_from 67 up_thru 46 down_at 64 last_clean_interval [60,63) [v2:127.0.0.1:6810/2980896430,v1:127.0.0.1:6811/2980896430] [v2:127.0.0.1:6812/2980896430,v1:127.0.0.1:6813/2980896430] exists,up 6d9c196a-4cd8-413e-bba5-8ecb673833b9 2026-02-13T02:03:16.957 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:03:16.957 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:03:16.957 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:03:16.957 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:03:16.957 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:03:16.957 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:03:16.957 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:03:16.957 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:03:16.957 INFO:tasks.workunit.client.0.trial112.stderr:129794: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:03:16.965 INFO:tasks.workunit.client.0.trial112.stderr:129794: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.very_ops 2026-02-13T02:03:16.965 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:03:16.965 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:03:16.965 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:03:16.965 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:03:16.965 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:03:16.965 INFO:tasks.workunit.client.0.trial112.stderr:129792: 2026-02-13T02:03:15.488+0000 7ff6e9946a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:16.965 INFO:tasks.workunit.client.0.trial112.stderr:129792: 2026-02-13T02:03:15.492+0000 7ff6e9946a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:16.965 INFO:tasks.workunit.client.0.trial112.stderr:129792: 2026-02-13T02:03:15.492+0000 7ff6e9946a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:16.965 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:03:16.966 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:03:16.966 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:03:16.966 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:03:16.966 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:03:16.966 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:03:16.966 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:03:16.966 INFO:tasks.workunit.client.0.trial112.stderr:129792: 0 2026-02-13T02:03:16.966 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:03:16.966 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:03:16.966 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:03:16.966 INFO:tasks.workunit.client.0.trial112.stderr:129792: 2026-02-13T02:03:15.968+0000 7ff6e9946a40 -1 Falling back to public interface 2026-02-13T02:03:16.966 INFO:tasks.workunit.client.0.trial112.stderr:129792: 2026-02-13T02:03:16.628+0000 7ff6e9946a40 -1 osd.1 63 log_to_monitors true 2026-02-13T02:03:16.966 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:03:16.967 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:03:16.967 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:03:16.967 INFO:tasks.workunit.client.0.trial112.stderr:129792: 1 2026-02-13T02:03:16.967 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:03:16.967 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:03:16.967 INFO:tasks.workunit.client.0.trial112.stderr:129792: osd.1 up in weight 1 up_from 67 up_thru 67 down_at 64 last_clean_interval [60,63) [v2:127.0.0.1:6818/2820817134,v1:127.0.0.1:6819/2820817134] [v2:127.0.0.1:6820/2820817134,v1:127.0.0.1:6821/2820817134] exists,up 776209f2-682b-43d3-a00a-d3186f312ec7 2026-02-13T02:03:16.967 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:03:16.967 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:03:16.967 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:03:16.967 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:03:16.967 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:03:16.967 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:03:16.967 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:03:16.968 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:03:16.968 INFO:tasks.workunit.client.0.trial112.stderr:129792: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:03:17.517 INFO:tasks.workunit.client.0.trial112.stderr:129792: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:03:17.518 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:03:17.518 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:03:17.518 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:03:17.518 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:03:17.518 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:03:17.518 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:03:17.518 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:03:17.518 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:03:17.518 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:03:17.519 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:03:17.519 INFO:tasks.workunit.client.0.trial112.stderr:129792: 1 2026-02-13T02:03:17.519 INFO:tasks.workunit.client.0.trial112.stderr:129792: 2 2026-02-13T02:03:17.519 INFO:tasks.workunit.client.0.trial112.stderr:129792: 3' 2026-02-13T02:03:17.519 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:03:17.519 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:17.520 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:03:17.520 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836503 2026-02-13T02:03:17.520 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836503 2026-02-13T02:03:17.520 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836503' 2026-02-13T02:03:17.520 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:17.520 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:03:17.520 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=287762808835 2026-02-13T02:03:17.520 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 287762808835 2026-02-13T02:03:17.521 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836503 1-287762808835' 2026-02-13T02:03:17.521 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:17.521 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:03:17.521 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=287762808834 2026-02-13T02:03:17.521 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 287762808834 2026-02-13T02:03:17.521 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836503 1-287762808835 2-287762808834' 2026-02-13T02:03:17.521 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:17.521 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:03:17.521 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=231928233991 2026-02-13T02:03:17.521 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 231928233991 2026-02-13T02:03:17.521 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836503 1-287762808835 2-287762808834 3-231928233991' 2026-02-13T02:03:17.521 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:17.522 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836503 2026-02-13T02:03:17.522 INFO:tasks.workunit.client.0.trial112.stderr:12979sh:1593: get_timeout_delays: echo true 2026-02-13T02:03:17.522 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:03:17.522 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:03:17.522 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:03:17.522 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:03:17.523 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:03:17.523 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:03:17.523 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:03:17.523 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:03:17.523 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:03:17.523 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:03:17.523 INFO:tasks.workunit.client.0.trial112.stderr:129794: 1 2026-02-13T02:03:17.523 INFO:tasks.workunit.client.0.trial112.stderr:129794: 2 2026-02-13T02:03:17.523 INFO:tasks.workunit.client.0.trial112.stderr:129794: 3' 2026-02-13T02:03:17.523 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:03:17.523 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:17.523 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:03:17.524 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836502 2026-02-13T02:03:17.524 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836502 2026-02-13T02:03:17.524 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502' 2026-02-13T02:03:17.524 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:17.524 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:03:17.524 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=287762808834 2026-02-13T02:03:17.524 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 287762808834 2026-02-13T02:03:17.524 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-287762808834' 2026-02-13T02:03:17.524 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:17.524 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:03:17.524 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=287762808835 2026-02-13T02:03:17.524 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 287762808835 2026-02-13T02:03:17.525 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-287762808834 2-287762808835' 2026-02-13T02:03:17.525 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:17.525 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:03:17.525 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=231928233992 2026-02-13T02:03:17.525 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 231928233992 2026-02-13T02:03:17.525 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-287762808834 2-287762808835 3-231928233992' 2026-02-13T02:03:17.525 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:17.525 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836502 2026-02-13T02:03:20.185 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:20.185 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:03:20.185 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836502 2026-02-13T02:03:20.185 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:20.185 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836502 2026-02-13T02:03:20.186 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2026-02-13T02:03:20.186 INFO:tasks.workunit.client.0.trial112.stderr:129794: waiting osd.0 seq 21474836502 2026-02-13T02:03:20.186 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:03:20.186 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836501 -lt 21474836502 2026-02-13T02:03:20.186 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:03:20.186 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:03:20.186 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:03:20.186 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836501 -lt 21474836502 2026-02-13T02:03:20.186 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:03:20.186 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:03:20.186 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:03:20.187 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836503 -lt 21474836502 2026-02-13T02:03:20.187 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:20.187 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-287762808834 2026-02-13T02:03:20.187 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:20.187 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:03:20.187 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-287762808834 2026-02-13T02:03:20.187 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:20.187 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=287762808834 2026-02-13T02:03:20.187 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 287762808834' 2026-02-13T02:03:20.187 INFO:tasks.workunit.client.0.trial112.stderr:129794: waiting osd.1 seq 287762808834 2026-02-13T02:03:20.187 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:03:20.188 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 287762808835 -lt 287762808834 2026-02-13T02:03:20.188 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:20.188 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-287762808835 2026-02-13T02:03:20.188 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:20.188 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:03:20.188 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-287762808835 2026-02-13T02:03:20.188 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:20.188 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=287762808835 2026-02-13T02:03:20.191 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 287762808832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:20.191 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:03:20.191 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836503 2026-02-13T02:03:20.191 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:20.191 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836503 2026-02-13T02:03:20.191 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2026-02-13T02:03:20.191 INFO:tasks.workunit.client.0.trial112.stderr:129792: waiting osd.0 seq 21474836503 2026-02-13T02:03:20.191 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:03:20.191 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836501 -lt 21474836503 2026-02-13T02:03:20.191 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:03:20.191 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:03:20.192 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:03:20.192 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836501 -lt 21474836503 2026-02-13T02:03:20.192 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:03:20.192 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:03:20.192 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:03:20.192 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836503 -lt 21474836503 2026-02-13T02:03:20.192 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:20.192 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-287762808835 2026-02-13T02:03:20.192 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:20.192 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:03:20.192 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-287762808835 2026-02-13T02:03:20.192 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:20.193 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=287762808835 2026-02-13T02:03:20.193 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 287762808835' 2026-02-13T02:03:20.193 INFO:tasks.workunit.client.0.trial112.stderr:129792: waiting osd.1 seq 287762808835 2026-02-13T02:03:20.193 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:03:20.193 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 287762808835 -lt 287762808835 2026-02-13T02:03:20.193 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:20.193 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-287762808834 2026-02-13T02:03:20.193 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:20.193 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:03:20.193 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-287762808834 2026-02-13T02:03:20.193 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:20.193 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=287762808834 2026-02-13T02:03:21.078 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 287762808835' 2026-02-13T02:03:21.078 INFO:tasks.workunit.client.0.trial112.stderr:129794: waiting osd.2 seq 287762808835 2026-02-13T02:03:21.078 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:03:21.078 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 287762808835 -lt 287762808835 2026-02-13T02:03:21.078 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:21.079 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-231928233992 2026-02-13T02:03:21.079 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:21.079 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:03:21.079 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-231928233992 2026-02-13T02:03:21.079 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:21.079 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=231928233992 2026-02-13T02:03:21.079 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 231928233992' 2026-02-13T02:03:21.079 INFO:tasks.workunit.client.0.trial112.stderr:129794: waiting osd.3 seq 231928233992 2026-02-13T02:03:21.079 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:03:21.079 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 231928233992 -lt 231928233992 2026-02-13T02:03:21.079 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:03:21.080 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:03:21.080 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:03:21.080 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:03:21.080 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:03:21.080 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:03:21.080 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:03:21.080 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:03:21.080 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:03:21.080 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:03:21.080 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:03:21.080 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:03:21.081 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:03:21.081 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:03:21.081 INFO:tasks.workunit.client.0.trial112.stderr:129794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:03:21.081 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:03:21.081 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:03:21.081 INFO:tasks.workunit.client.0.trial112.stderr:129794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:03:21.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-13T02:03:21.082 INFO:tasks.workunit.client.0.trial112.stderr:4' 2026-02-13T02:03:21.083 INFO:tasks.workunit.client.0.trial112.stderr:129792: waiting osd.2 seq 287762808834 2026-02-13T02:03:21.083 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:03:21.083 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 287762808835 -lt 287762808834 2026-02-13T02:03:21.083 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:21.083 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-231928233991 2026-02-13T02:03:21.083 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:21.083 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:03:21.083 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-231928233991 2026-02-13T02:03:21.083 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:21.083 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=231928233991 2026-02-13T02:03:21.083 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 231928233991' 2026-02-13T02:03:21.084 INFO:tasks.workunit.client.0.trial112.stderr:129792: waiting osd.3 seq 231928233991 2026-02-13T02:03:21.084 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:03:21.084 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 231928233992 -lt 231928233991 2026-02-13T02:03:21.084 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:03:21.084 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:03:21.084 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:03:21.084 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:03:21.084 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:03:21.084 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:03:21.084 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:03:21.084 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:03:21.084 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:03:21.085 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:03:21.085 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:03:21.085 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:03:21.085 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:03:21.085 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:03:21.085 INFO:tasks.workunit.client.0.trial112.stderr:129792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:03:21.085 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:03:21.085 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:03:21.085 INFO:tasks.workunit.client.0.trial112.stderr:129792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:03:21.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-13T02:03:21.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-13T02:03:21.086 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 129791 2026-02-13T02:03:21.086 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-13T02:03:21.086 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-13T02:03:21.086 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-13T02:03:21.086 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:272: corrupt_and_repair_two: return_code=0 2026-02-13T02:03:21.086 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-13T02:03:21.086 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:275: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-13T02:03:21.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:276: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-13T02:03:21.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:323: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2026-02-13T02:03:21.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:245: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2026-02-13T02:03:21.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:246: corrupt_and_repair_two: local poolname=ecpool 2026-02-13T02:03:21.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:247: corrupt_and_repair_two: local first=3 2026-02-13T02:03:21.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_two: local second=1 2026-02-13T02:03:21.104 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_two: pids= 2026-02-13T02:03:21.104 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-13T02:03:21.104 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-13T02:03:21.104 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-13T02:03:21.104 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 132108"' 2026-02-13T02:03:21.104 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 132108' 2026-02-13T02:03:21.104 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2026-02-13T02:03:21.104 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-13T02:03:21.104 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-13T02:03:21.104 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 132110"' 2026-02-13T02:03:21.104 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 132110' 2026-02-13T02:03:21.105 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_two: wait_background pids 2026-02-13T02:03:21.105 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-13T02:03:21.105 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 132108 132110' 2026-02-13T02:03:21.105 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-13T02:03:21.105 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-13T02:03:21.105 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 132108 2026-02-13T02:03:21.105 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/132111: /' 2026-02-13T02:03:21.105 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2026-02-13T02:03:21.105 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/132113: /' 2026-02-13T02:03:21.748 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:03:21.749 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:03:21.749 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-13T02:03:21.749 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:03:21.749 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2026-02-13T02:03:21.749 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:03:21.749 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:03:21.749 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-13T02:03:21.749 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:03:21.749 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-13T02:03:21.750 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:03:21.750 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:03:21.750 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:03:21.750 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:03:21.750 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:03:21.750 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:03:21.750 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2026-02-13T02:03:21.750 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:03:21.750 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:03:21.750 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-13T02:03:21.751 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:03:21.751 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:03:21.751 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2026-02-13T02:03:21.751 INFO:tasks.workunit.client.0.trial112.stderr:132111: remove 0#2:eb822e21:::SOMETHING:head# 2026-02-13T02:03:21.751 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-13T02:03:21.751 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:03:21.751 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:03:21.751 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-13T02:03:21.751 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:03:21.751 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:03:21.751 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:03:21.752 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:03:21.752 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:03:21.752 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:03:21.752 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-13T02:03:21.753 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-13T02:03:21.753 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:03:21.753 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:03:21.753 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:03:21.753 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:03:21.753 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:03:21.753 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:03:21.753 INFO:tasks.workunit.client.0.trial112.stderr:132111: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:03:21.753 INFO:tasks.workunit.client.0.trial112.stderr:132111: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:03:21.753 INFO:tasks.workunit.client.0.trial112.stderr:132111: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:03:21.754 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:03:21.754 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:03:21.754 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:03:21.754 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:03:21.754 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:03:21.754 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:03:21.754 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:03:21.754 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:03:21.754 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:03:21.754 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:03:21.754 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:03:21.755 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-13T02:03:21.755 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-13T02:03:21.755 INFO:tasks.workunit.client.0.trial112.stderr:132111: start osd.3 2026-02-13T02:03:21.755 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:03:21.784 INFO:tasks.workunit.client.0.trial112.stderr:1132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:03:21.785 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:03:21.785 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:03:21.785 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:03:21.785 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2026-02-13T02:03:21.785 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:03:21.785 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:03:21.785 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:03:21.785 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:03:21.785 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:03:21.785 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:03:21.785 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:03:21.786 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:03:21.786 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:03:21.786 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:03:21.786 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:03:21.786 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2026-02-13T02:03:21.786 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:03:21.786 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:03:21.786 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:03:21.786 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:03:21.786 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:03:21.786 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2026-02-13T02:03:21.786 INFO:tasks.workunit.client.0.trial112.stderr:132113: remove 1#2:eb822e21:::SOMETHING:head# 2026-02-13T02:03:21.787 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-13T02:03:21.787 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:03:21.787 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:03:21.787 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:03:21.787 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:03:21.787 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:03:21.787 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:03:21.787 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:03:21.788 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:03:21.788 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:03:21.788 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:03:21.788 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:03:21.788 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:03:21.788 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:03:21.788 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:03:21.788 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:03:21.788 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:03:21.789 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:03:21.789 INFO:tasks.workunit.client.0.trial112.stderr:132113: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:03:21.789 INFO:tasks.workunit.client.0.trial112.stderr:132113: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:03:21.789 INFO:tasks.workunit.client.0.trial112.stderr:132113: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:03:21.789 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:03:21.789 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:03:21.789 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:03:21.789 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:03:21.789 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:03:21.789 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:03:21.789 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:03:21.789 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:03:21.789 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:03:21.790 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:03:21.790 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:03:21.790 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:03:21.790 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:03:21.790 INFO:tasks.workunit.client.0.trial112.stderr:132113: start osd.1 2026-02-13T02:03:21.790 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:03:23.236 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-13T02:03:23.237 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-13T02:03:23.237 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:03:23.237 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:03:23.237 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:03:23.237 INFO:tasks.workunit.client.0.trial112.stderr:132111: 2026-02-13T02:03:21.760+0000 7fa0709e2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:23.237 INFO:tasks.workunit.client.0.trial112.stderr:132111: 2026-02-13T02:03:21.764+0000 7fa0709e2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:23.237 INFO:tasks.workunit.client.0.trial112.stderr:132111: 2026-02-13T02:03:21.764+0000 7fa0709e2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:23.238 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-13T02:03:23.238 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:03:23.238 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-13T02:03:23.238 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:03:23.238 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:03:23.238 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:03:23.238 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:03:23.238 INFO:tasks.workunit.client.0.trial112.stderr:132111: 0 2026-02-13T02:03:23.238 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:03:23.238 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:03:23.239 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:03:23.239 INFO:tasks.workunit.client.0.trial112.stderr:132111: 2026-02-13T02:03:22.236+0000 7fa0709e2a40 -1 Falling back to public interface 2026-02-13T02:03:23.239 INFO:tasks.workunit.client.0.trial112.stderr:132111: 2026-02-13T02:03:22.888+0000 7fa0709e2a40 -1 osd.3 68 log_to_monitors true 2026-02-13T02:03:23.239 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:03:23.239 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:03:23.239 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:03:23.239 INFO:tasks.workunit.client.0.trial112.stderr:132111: 1 2026-02-13T02:03:23.239 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:03:23.239 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:03:23.239 INFO:tasks.workunit.client.0.trial112.stderr:132111: osd.3 up in weight 1 up_from 73 up_thru 73 down_at 69 last_clean_interval [54,68) [v2:127.0.0.1:6818/3604449369,v1:127.0.0.1:6819/3604449369] [v2:127.0.0.1:6820/3604449369,v1:127.0.0.1:6821/3604449369] exists,up 61ebaa55-733a-4e41-b1bf-de710ed831df 2026-02-13T02:03:23.239 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:03:23.239 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:03:23.240 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:03:23.240 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:03:23.240 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:03:23.240 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:03:23.240 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:03:23.240 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:03:23.240 INFO:tasks.workunit.client.0.trial112.stderr:132111: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:03:23.282 INFO:tasks.workunit.client.0.trial112.stderr:132111: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: g32113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:03:23.282 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:03:23.283 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:03:23.283 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:03:23.283 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:03:23.283 INFO:tasks.workunit.client.0.trial112.stderr:132113: 2026-02-13T02:03:21.792+0000 7f2f12a75a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:23.283 INFO:tasks.workunit.client.0.trial112.stderr:132113: 2026-02-13T02:03:21.792+0000 7f2f12a75a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:23.283 INFO:tasks.workunit.client.0.trial112.stderr:132113: 2026-02-13T02:03:21.796+0000 7f2f12a75a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:23.283 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:03:23.283 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:03:23.283 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:03:23.283 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:03:23.283 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:03:23.284 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:03:23.284 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:03:23.284 INFO:tasks.workunit.client.0.trial112.stderr:132113: 0 2026-02-13T02:03:23.284 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:03:23.284 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:03:23.284 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:03:23.284 INFO:tasks.workunit.client.0.trial112.stderr:132113: 2026-02-13T02:03:22.248+0000 7f2f12a75a40 -1 Falling back to public interface 2026-02-13T02:03:23.284 INFO:tasks.workunit.client.0.trial112.stderr:132113: 2026-02-13T02:03:22.932+0000 7f2f12a75a40 -1 osd.1 68 log_to_monitors true 2026-02-13T02:03:23.284 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:03:23.284 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:03:23.284 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:03:23.284 INFO:tasks.workunit.client.0.trial112.stderr:132113: 1 2026-02-13T02:03:23.285 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:03:23.285 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:03:23.285 INFO:tasks.workunit.client.0.trial112.stderr:132113: osd.1 up in weight 1 up_from 73 up_thru 73 down_at 69 last_clean_interval [67,68) [v2:127.0.0.1:6826/2216859425,v1:127.0.0.1:6827/2216859425] [v2:127.0.0.1:6828/2216859425,v1:127.0.0.1:6829/2216859425] exists,up 776209f2-682b-43d3-a00a-d3186f312ec7 2026-02-13T02:03:23.285 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:03:23.285 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:03:23.285 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:03:23.285 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:03:23.285 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:03:23.285 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:03:23.285 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:03:23.285 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:03:23.285 INFO:tasks.workunit.client.0.trial112.stderr:132113: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:03:23.805 INFO:tasks.workunit.client.0.trial112.stderr:132113: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:03:23.805 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:03:23.805 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:03:23.805 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:03:23.805 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:03:23.805 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:03:23.805 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:03:23.805 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:03:23.806 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:03:23.806 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:03:23.806 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:03:23.806 INFO:tasks.workunit.client.0.trial112.stderr:132111: 1 2026-02-13T02:03:23.806 INFO:tasks.workunit.client.0.trial112.stderr:132111: 2 2026-02-13T02:03:23.806 INFO:tasks.workunit.client.0.trial112.stderr:132111: 3' 2026-02-13T02:03:23.806 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:03:23.806 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:23.806 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:03:23.806 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836505 2026-02-13T02:03:23.806 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836505 2026-02-13T02:03:23.806 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505' 2026-02-13T02:03:23.806 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:23.806 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:03:23.806 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=313532612610 2026-02-13T02:03:23.806 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 313532612610 2026-02-13T02:03:23.806 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-313532612610' 2026-02-13T02:03:23.806 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:23.806 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:03:23.806 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=287762808837 2026-02-13T02:03:23.806 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 287762808837 2026-02-13T02:03:23.806 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-313532612610 2-287762808837' 2026-02-13T02:03:23.806 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:23.806 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:03:23.806 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=313532612610 2026-02-13T02:03:23.806 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 313532612610 2026-02-13T02:03:23.806 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-313532612610 2-287762808837 3-313532612610' 2026-02-13T02:03:23.806 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:23.806 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836505 2026-02-13T02:03:23.836 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/et_timeout_delays: echo true 2026-02-13T02:03:23.836 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:03:23.836 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:03:23.837 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:03:23.837 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:03:23.837 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:03:23.837 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:03:23.837 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:03:23.837 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:03:23.837 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:03:23.837 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:03:23.837 INFO:tasks.workunit.client.0.trial112.stderr:132113: 1 2026-02-13T02:03:23.837 INFO:tasks.workunit.client.0.trial112.stderr:132113: 2 2026-02-13T02:03:23.837 INFO:tasks.workunit.client.0.trial112.stderr:132113: 3' 2026-02-13T02:03:23.837 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:03:23.837 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:23.837 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:03:23.837 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836506 2026-02-13T02:03:23.837 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836506 2026-02-13T02:03:23.837 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836506' 2026-02-13T02:03:23.837 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:23.837 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:03:23.837 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=313532612611 2026-02-13T02:03:23.837 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 313532612611 2026-02-13T02:03:23.837 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836506 1-313532612611' 2026-02-13T02:03:23.837 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:23.837 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:03:23.837 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=287762808838 2026-02-13T02:03:23.837 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 287762808838 2026-02-13T02:03:23.837 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836506 1-313532612611 2-287762808838' 2026-02-13T02:03:23.837 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:23.837 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:03:23.837 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=313532612611 2026-02-13T02:03:23.837 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 313532612611 2026-02-13T02:03:23.837 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836506 1-313532612611 2-287762808838 3-313532612611' 2026-02-13T02:03:23.837 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:23.838 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836506 2026-02-13T02:03:25.469 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:25.469 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:03:25.470 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836505 2026-02-13T02:03:25.470 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:25.470 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836505 2026-02-13T02:03:25.470 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2026-02-13T02:03:25.470 INFO:tasks.workunit.client.0.trial112.stderr:132111: waiting osd.0 seq 21474836505 2026-02-13T02:03:25.470 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:03:25.470 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836504 -lt 21474836505 2026-02-13T02:03:25.470 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:03:25.470 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:03:25.470 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:03:25.471 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836507 -lt 21474836505 2026-02-13T02:03:25.471 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:25.471 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-313532612610 2026-02-13T02:03:25.471 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:25.471 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:03:25.471 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-313532612610 2026-02-13T02:03:25.471 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:25.471 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=313532612610 2026-02-13T02:03:25.471 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 313532612610' 2026-02-13T02:03:25.471 INFO:tasks.workunit.client.0.trial112.stderr:132111: waiting osd.1 seq 313532612610 2026-02-13T02:03:25.471 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:03:25.471 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 313532612611 -lt 313532612610 2026-02-13T02:03:25.472 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:25.472 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-287762808837 2026-02-13T02:03:25.472 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:25.472 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:03:25.472 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-287762808837 2026-02-13T02:03:25.472 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:25.472 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=287762808837 2026-02-13T02:03:25.472 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 287762808837' 2026-02-13T02:03:25.472 INFO:tasks.workunit.client.0.trial112.stderr:132111: waiting osd.2 seq 287762808837 2026-02-13T02:03:25.472 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:03:25.472 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 287762808838 -lt 287762808837 2026-02-13T02:03:25.472 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:25.500 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.shubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:25.500 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:03:25.500 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836506 2026-02-13T02:03:25.500 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:25.500 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836506 2026-02-13T02:03:25.500 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2026-02-13T02:03:25.500 INFO:tasks.workunit.client.0.trial112.stderr:132113: waiting osd.0 seq 21474836506 2026-02-13T02:03:25.500 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:03:25.500 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836504 -lt 21474836506 2026-02-13T02:03:25.501 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:03:25.501 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:03:25.501 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:03:25.501 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836507 -lt 21474836506 2026-02-13T02:03:25.501 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:25.501 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-313532612611 2026-02-13T02:03:25.501 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:25.501 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:03:25.501 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-313532612611 2026-02-13T02:03:25.501 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:25.501 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=313532612611 2026-02-13T02:03:25.501 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 313532612611' 2026-02-13T02:03:25.501 INFO:tasks.workunit.client.0.trial112.stderr:132113: waiting osd.1 seq 313532612611 2026-02-13T02:03:25.502 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:03:25.502 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 313532612611 -lt 313532612611 2026-02-13T02:03:25.502 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:25.502 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-287762808838 2026-02-13T02:03:25.502 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:25.502 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:03:25.502 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-287762808838 2026-02-13T02:03:25.502 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:25.502 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=287762808838 2026-02-13T02:03:25.502 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 287762808838' 2026-02-13T02:03:25.502 INFO:tasks.workunit.client.0.trial112.stderr:132113: waiting osd.2 seq 287762808838 2026-02-13T02:03:25.502 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:03:25.503 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 287762808838 -lt 287762808838 2026-02-13T02:03:25.503 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:26.188 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-313532612610 2026-02-13T02:03:26.189 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:26.189 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:03:26.189 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-313532612610 2026-02-13T02:03:26.189 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:26.189 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=313532612610 2026-02-13T02:03:26.189 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 313532612610' 2026-02-13T02:03:26.189 INFO:tasks.workunit.client.0.trial112.stderr:132111: waiting osd.3 seq 313532612610 2026-02-13T02:03:26.189 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:03:26.189 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 313532612611 -lt 313532612610 2026-02-13T02:03:26.190 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:03:26.190 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:03:26.190 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:03:26.190 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:03:26.190 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:03:26.190 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:03:26.190 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:03:26.190 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:03:26.190 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:03:26.190 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:03:26.190 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:03:26.190 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:03:26.190 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:03:26.191 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:03:26.191 INFO:tasks.workunit.client.0.trial112.stderr:132111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:03:26.191 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:03:26.191 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:03:26.191 INFO:tasks.workunit.client.0.trial112.stderr:132111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:03:26.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-13T02:03:26.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-13T02:03:26.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 132110 2026-02-13T02:03:26.233 INFO:tasks.workunit.client.0.trial112.stderr::2267: flush_pg_stats: echo 3-313532612611 2026-02-13T02:03:26.234 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:26.234 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:03:26.234 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-313532612611 2026-02-13T02:03:26.234 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:26.234 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=313532612611 2026-02-13T02:03:26.234 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 313532612611' 2026-02-13T02:03:26.234 INFO:tasks.workunit.client.0.trial112.stderr:132113: waiting osd.3 seq 313532612611 2026-02-13T02:03:26.234 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:03:26.234 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 313532612611 -lt 313532612611 2026-02-13T02:03:26.235 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:03:26.235 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:03:26.235 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:03:26.235 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:03:26.235 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:03:26.235 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:03:26.235 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:03:26.235 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:03:26.235 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:03:26.235 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:03:26.235 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:03:26.235 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:03:26.236 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:03:26.236 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:03:26.236 INFO:tasks.workunit.client.0.trial112.stderr:132113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:03:26.236 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:03:26.236 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:03:26.236 INFO:tasks.workunit.client.0.trial112.stderr:132113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:03:26.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-13T02:03:26.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-13T02:03:26.238 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-13T02:03:26.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-13T02:03:26.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:257: corrupt_and_repair_two: return_code=0 2026-02-13T02:03:26.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:258: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-13T02:03:26.238 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: get_pg ecpool SOMETHING 2026-02-13T02:03:26.238 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-13T02:03:26.238 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-13T02:03:26.238 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-13T02:03:26.238 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T02:03:26.401 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: local pg=2.0 2026-02-13T02:03:26.401 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:264: corrupt_and_repair_two: repair 2.0 2026-02-13T02:03:26.402 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-13T02:03:26.402 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-13T02:03:26.402 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:03:26.402 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:03:26.402 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:03:26.402 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:03:26.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-13T02:03:14.427998+0000 2026-02-13T02:03:26.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-13T02:03:26.716 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-13T02:03:26.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-13T02:03:14.427998+0000 2026-02-13T02:03:26.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-13T02:03:26.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:03:14.427998+0000 2026-02-13T02:03:26.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:03:26.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:03:26.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:03:26.728 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:03:26.728 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:03:26.728 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:03:26.728 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:03:26.728 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:03:26.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:03:14.427998+0000 '>' 2026-02-13T02:03:14.427998+0000 2026-02-13T02:03:26.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:03:27.892 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:03:27.892 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:03:27.893 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:03:27.893 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:03:27.893 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:03:27.893 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:03:27.893 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:03:28.052 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:03:14.427998+0000 '>' 2026-02-13T02:03:14.427998+0000 2026-02-13T02:03:28.053 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:03:29.054 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:03:29.054 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:03:29.055 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:03:29.055 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:03:29.055 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:03:29.055 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:03:29.055 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:03:29.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:03:26.821513+0000 '>' 2026-02-13T02:03:14.427998+0000 2026-02-13T02:03:29.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:03:29.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: corrupt_and_repair_two: pids= 2026-02-13T02:03:29.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-13T02:03:29.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-13T02:03:29.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-13T02:03:29.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 134870"' 2026-02-13T02:03:29.215 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 134870' 2026-02-13T02:03:29.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-13T02:03:29.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-13T02:03:29.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-13T02:03:29.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 134872"' 2026-02-13T02:03:29.215 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 134872' 2026-02-13T02:03:29.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-13T02:03:29.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:271: corrupt_and_repair_two: wait_background pids 2026-02-13T02:03:29.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 134870 134872' 2026-02-13T02:03:29.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-13T02:03:29.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-13T02:03:29.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 134870 2026-02-13T02:03:29.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/134873: /' 2026-02-13T02:03:29.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-13T02:03:29.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/134875: /' 2026-02-13T02:03:29.700 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:03:29.700 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:03:29.700 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-13T02:03:29.700 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:03:29.700 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-13T02:03:29.700 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:03:29.701 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:03:29.701 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-13T02:03:29.701 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:03:29.701 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-13T02:03:29.701 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:03:29.701 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:03:29.701 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:03:29.701 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:03:29.701 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:03:29.701 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:03:29.701 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-13T02:03:29.701 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:03:29.702 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:03:29.702 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-13T02:03:29.702 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:03:29.702 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:03:29.702 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2026-02-13T02:03:29.702 INFO:tasks.workunit.client.0.trial112.stderr:134873: Error getting attr on : 2.0s0_head,0#-4:00000000:::deep_scrub_2.0s0:head#, (61) No data available 2026-02-13T02:03:29.702 INFO:tasks.workunit.client.0.trial112.stderr:134873: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2026-02-13T02:03:29.702 INFO:tasks.workunit.client.0.trial112.stderr:134873: _ 2026-02-13T02:03:29.702 INFO:tasks.workunit.client.0.trial112.stderr:134873: hinfo_key 2026-02-13T02:03:29.702 INFO:tasks.workunit.client.0.trial112.stderr:134873: snapset 2026-02-13T02:03:29.702 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-13T02:03:29.702 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:03:29.703 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:03:29.703 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-13T02:03:29.703 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:03:29.703 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:03:29.703 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:03:29.703 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:03:29.704 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:03:29.704 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:03:29.704 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-13T02:03:29.704 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-13T02:03:29.704 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:03:29.704 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:03:29.704 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:03:29.704 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:03:29.704 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:03:29.704 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:03:29.704 INFO:tasks.workunit.client.0.trial112.stderr:134873: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:03:29.705 INFO:tasks.workunit.client.0.trial112.stderr:134873: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:03:29.705 INFO:tasks.workunit.client.0.trial112.stderr:134873: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:03:29.705 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:03:29.705 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:03:29.705 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:03:29.705 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:03:29.705 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:03:29.705 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:03:29.705 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:03:29.705 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:03:29.705 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:03:29.705 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:03:29.705 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:03:29.706 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-13T02:03:29.706 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-13T02:03:29.706 INFO:tasks.workunit.client.0.trial112.stderr:134873: start osd.3 2026-02-13T02:03:29.716 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:03:29.716 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:03:29.716 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:03:29.716 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:03:29.716 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-13T02:03:29.716 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:03:29.717 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:03:29.717 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:03:29.717 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:03:29.717 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:03:29.717 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:03:29.717 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:03:29.717 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:03:29.717 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:03:29.717 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:03:29.717 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:03:29.717 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-13T02:03:29.717 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:03:29.718 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:03:29.718 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:03:29.718 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:03:29.718 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:03:29.718 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2026-02-13T02:03:29.718 INFO:tasks.workunit.client.0.trial112.stderr:134875: _ 2026-02-13T02:03:29.718 INFO:tasks.workunit.client.0.trial112.stderr:134875: hinfo_key 2026-02-13T02:03:29.718 INFO:tasks.workunit.client.0.trial112.stderr:134875: snapset 2026-02-13T02:03:29.718 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-13T02:03:29.718 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:03:29.718 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:03:29.718 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:03:29.718 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:03:29.719 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:03:29.719 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:03:29.719 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:03:29.719 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:03:29.720 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:03:29.720 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:03:29.720 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:03:29.720 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:03:29.720 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:03:29.720 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:03:29.720 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:03:29.720 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:03:29.720 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:03:29.720 INFO:tasks.workunit.client.0.trial112.stderr:134875: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:03:29.720 INFO:tasks.workunit.client.0.trial112.stderr:134875: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:03:29.721 INFO:tasks.workunit.client.0.trial112.stderr:134875: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:03:29.721 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:03:29.721 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:03:29.721 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:03:29.721 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:03:29.721 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:03:29.721 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:03:29.721 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:03:29.721 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:03:29.721 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:03:29.721 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:03:29.721 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:03:29.722 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:03:29.722 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:03:29.722 INFO:tasks.workunit.client.0.trial112.stderr:134875: start osd.1 2026-02-13T02:03:31.201 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recolog' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:03:31.202 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-13T02:03:31.202 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-13T02:03:31.202 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:03:31.202 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:03:31.202 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:03:31.202 INFO:tasks.workunit.client.0.trial112.stderr:134873: 2026-02-13T02:03:29.712+0000 7f314331fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:31.202 INFO:tasks.workunit.client.0.trial112.stderr:134873: 2026-02-13T02:03:29.712+0000 7f314331fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:31.202 INFO:tasks.workunit.client.0.trial112.stderr:134873: 2026-02-13T02:03:29.716+0000 7f314331fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:31.202 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-13T02:03:31.202 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:03:31.203 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-13T02:03:31.203 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:03:31.203 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:03:31.203 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:03:31.203 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:03:31.203 INFO:tasks.workunit.client.0.trial112.stderr:134873: 0 2026-02-13T02:03:31.203 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:03:31.203 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:03:31.203 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:03:31.203 INFO:tasks.workunit.client.0.trial112.stderr:134873: 2026-02-13T02:03:30.196+0000 7f314331fa40 -1 Falling back to public interface 2026-02-13T02:03:31.203 INFO:tasks.workunit.client.0.trial112.stderr:134873: 2026-02-13T02:03:30.828+0000 7f314331fa40 -1 osd.3 75 log_to_monitors true 2026-02-13T02:03:31.203 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:03:31.203 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:03:31.204 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:03:31.204 INFO:tasks.workunit.client.0.trial112.stderr:134873: 1 2026-02-13T02:03:31.204 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:03:31.204 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:03:31.204 INFO:tasks.workunit.client.0.trial112.stderr:134873: osd.3 up in weight 1 up_from 79 up_thru 79 down_at 76 last_clean_interval [73,75) [v2:127.0.0.1:6818/3978079499,v1:127.0.0.1:6819/3978079499] [v2:127.0.0.1:6820/3978079499,v1:127.0.0.1:6821/3978079499] exists,up 61ebaa55-733a-4e41-b1bf-de710ed831df 2026-02-13T02:03:31.204 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:03:31.204 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:03:31.204 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:03:31.204 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:03:31.204 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:03:31.204 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:03:31.204 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:03:31.211 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delvery_ops 2026-02-13T02:03:31.211 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:03:31.211 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:03:31.211 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:03:31.211 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:03:31.211 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:03:31.211 INFO:tasks.workunit.client.0.trial112.stderr:134875: 2026-02-13T02:03:29.724+0000 7fb143296a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:31.211 INFO:tasks.workunit.client.0.trial112.stderr:134875: 2026-02-13T02:03:29.728+0000 7fb143296a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:31.212 INFO:tasks.workunit.client.0.trial112.stderr:134875: 2026-02-13T02:03:29.728+0000 7fb143296a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:31.212 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:03:31.212 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:03:31.212 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:03:31.212 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:03:31.212 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:03:31.212 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:03:31.212 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:03:31.212 INFO:tasks.workunit.client.0.trial112.stderr:134875: 0 2026-02-13T02:03:31.212 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:03:31.212 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:03:31.213 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:03:31.213 INFO:tasks.workunit.client.0.trial112.stderr:134875: 2026-02-13T02:03:30.244+0000 7fb143296a40 -1 Falling back to public interface 2026-02-13T02:03:31.213 INFO:tasks.workunit.client.0.trial112.stderr:134875: 2026-02-13T02:03:30.916+0000 7fb143296a40 -1 osd.1 75 log_to_monitors true 2026-02-13T02:03:31.213 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:03:31.213 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:03:31.213 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:03:31.213 INFO:tasks.workunit.client.0.trial112.stderr:134875: 1 2026-02-13T02:03:31.213 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:03:31.213 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:03:31.213 INFO:tasks.workunit.client.0.trial112.stderr:134875: osd.1 up in weight 1 up_from 80 up_thru 80 down_at 76 last_clean_interval [73,75) [v2:127.0.0.1:6826/167917140,v1:127.0.0.1:6827/167917140] [v2:127.0.0.1:6828/167917140,v1:127.0.0.1:6829/167917140] exists,up 776209f2-682b-43d3-a00a-d3186f312ec7 2026-02-13T02:03:31.213 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:03:31.213 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:03:31.213 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:03:31.214 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:03:31.214 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:03:31.214 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:03:31.214 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:03:31.214 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:03:31.214 INFO:tasks.workunit.client.0.trial112.stderr:134875: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:03:31.775 INFO:tasks.workunit.client.0.trial112.stderr:134875: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:03:31.776 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:03:31.776 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:03:31.776 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:03:31.776 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:03:31.776 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:03:31.776 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:03:31.776 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:03:31.776 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:03:31.776 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:03:31.777 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:03:31.777 INFO:tasks.workunit.client.0.trial112.stderr:134875: 1 2026-02-13T02:03:31.777 INFO:tasks.workunit.client.0.trial112.stderr:134875: 2 2026-02-13T02:03:31.777 INFO:tasks.workunit.client.0.trial112.stderr:134875: 3' 2026-02-13T02:03:31.777 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:03:31.777 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:31.777 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:03:31.777 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836510 2026-02-13T02:03:31.777 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836510 2026-02-13T02:03:31.777 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510' 2026-02-13T02:03:31.777 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:31.777 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:03:31.778 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=343597383682 2026-02-13T02:03:31.778 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 343597383682 2026-02-13T02:03:31.778 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-343597383682' 2026-02-13T02:03:31.778 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:31.778 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:03:31.778 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=287762808840 2026-02-13T02:03:31.778 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 287762808840 2026-02-13T02:03:31.778 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-343597383682 2-287762808840' 2026-02-13T02:03:31.778 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:31.778 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:03:31.778 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=339302416386 2026-02-13T02:03:31.778 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 339302416386 2026-02-13T02:03:31.778 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-343597383682 2-287762808840 3-339302416386' 2026-02-13T02:03:31.779 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:31.779 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836510 2026-02-13T02:03:31.779 INFO:tasks.workunit.client.0.trial112.stderr:134875: /ays 90 .1 2026-02-13T02:03:31.779 INFO:tasks.workunit.client.0.trial112.stderr:134873: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:03:31.779 INFO:tasks.workunit.client.0.trial112.stderr:134873: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:03:31.779 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:03:31.780 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:03:31.780 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:03:31.780 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:03:31.780 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:03:31.780 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:03:31.780 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:03:31.780 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:03:31.780 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:03:31.780 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:03:31.780 INFO:tasks.workunit.client.0.trial112.stderr:134873: 1 2026-02-13T02:03:31.780 INFO:tasks.workunit.client.0.trial112.stderr:134873: 2 2026-02-13T02:03:31.780 INFO:tasks.workunit.client.0.trial112.stderr:134873: 3' 2026-02-13T02:03:31.781 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:03:31.781 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:31.781 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:03:31.781 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836509 2026-02-13T02:03:31.781 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836509 2026-02-13T02:03:31.781 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509' 2026-02-13T02:03:31.781 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:31.781 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:03:31.781 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=343597383683 2026-02-13T02:03:31.781 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 343597383683 2026-02-13T02:03:31.781 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-343597383683' 2026-02-13T02:03:31.781 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:31.782 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:03:31.782 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=287762808841 2026-02-13T02:03:31.782 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 287762808841 2026-02-13T02:03:31.782 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-343597383683 2-287762808841' 2026-02-13T02:03:31.782 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:31.782 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:03:31.782 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=339302416387 2026-02-13T02:03:31.782 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 339302416387 2026-02-13T02:03:31.782 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-343597383683 2-287762808841 3-339302416387' 2026-02-13T02:03:33.443 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:33.443 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836509 2026-02-13T02:03:33.444 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:33.444 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:03:33.444 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836509 2026-02-13T02:03:33.444 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:33.444 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836509 2026-02-13T02:03:33.444 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2026-02-13T02:03:33.444 INFO:tasks.workunit.client.0.trial112.stderr:134873: waiting osd.0 seq 21474836509 2026-02-13T02:03:33.444 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:03:33.444 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836508 -lt 21474836509 2026-02-13T02:03:33.444 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:03:33.444 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:03:33.445 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:03:33.445 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836510 -lt 21474836509 2026-02-13T02:03:33.445 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:33.445 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-343597383683 2026-02-13T02:03:33.445 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:33.445 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:03:33.445 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-343597383683 2026-02-13T02:03:33.445 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:33.445 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=343597383683 2026-02-13T02:03:33.445 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 343597383683' 2026-02-13T02:03:33.445 INFO:tasks.workunit.client.0.trial112.stderr:134873: waiting osd.1 seq 343597383683 2026-02-13T02:03:33.445 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:03:33.446 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 343597383683 -lt 343597383683 2026-02-13T02:03:33.446 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:33.446 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-287762808841 2026-02-13T02:03:33.446 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:33.446 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:03:33.446 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-287762808841 2026-02-13T02:03:33.446 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:33.446 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=287762808841 2026-02-13T02:03:33.446 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 287762808841' 2026-02-13T02:03:33.446 INFO:tasks.workunit.client.0.trial112.stderr:134873: waiting osd.2 seq 287762808841 2026-02-13T02:03:33.446 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:03:33.448 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 2/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:33.448 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:03:33.448 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836510 2026-02-13T02:03:33.448 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:33.448 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836510 2026-02-13T02:03:33.448 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2026-02-13T02:03:33.448 INFO:tasks.workunit.client.0.trial112.stderr:134875: waiting osd.0 seq 21474836510 2026-02-13T02:03:33.448 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:03:33.448 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836508 -lt 21474836510 2026-02-13T02:03:33.448 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:03:33.449 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:03:33.449 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:03:33.449 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836510 -lt 21474836510 2026-02-13T02:03:33.449 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:33.449 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-343597383682 2026-02-13T02:03:33.449 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:33.449 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:03:33.449 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-343597383682 2026-02-13T02:03:33.449 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:33.449 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=343597383682 2026-02-13T02:03:33.449 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 343597383682' 2026-02-13T02:03:33.449 INFO:tasks.workunit.client.0.trial112.stderr:134875: waiting osd.1 seq 343597383682 2026-02-13T02:03:33.449 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:03:33.450 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 343597383683 -lt 343597383682 2026-02-13T02:03:33.450 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:33.450 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-287762808840 2026-02-13T02:03:33.450 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:33.450 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:03:33.450 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-287762808840 2026-02-13T02:03:33.450 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:33.450 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=287762808840 2026-02-13T02:03:33.450 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 287762808840' 2026-02-13T02:03:33.450 INFO:tasks.workunit.client.0.trial112.stderr:134875: waiting osd.2 seq 287762808840 2026-02-13T02:03:33.450 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:03:33.450 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 287762808842 -lt 287762808840 2026-02-13T02:03:33.450 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:34.171 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-help87762808842 -lt 287762808841 2026-02-13T02:03:34.172 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:34.172 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-339302416387 2026-02-13T02:03:34.172 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:34.172 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:03:34.172 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-339302416387 2026-02-13T02:03:34.172 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:34.172 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=339302416387 2026-02-13T02:03:34.172 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 339302416387' 2026-02-13T02:03:34.172 INFO:tasks.workunit.client.0.trial112.stderr:134873: waiting osd.3 seq 339302416387 2026-02-13T02:03:34.173 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:03:34.173 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 339302416387 -lt 339302416387 2026-02-13T02:03:34.173 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:03:34.173 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:03:34.173 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:03:34.173 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:03:34.173 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:03:34.173 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:03:34.173 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:03:34.173 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:03:34.173 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:03:34.173 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:03:34.174 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:03:34.174 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:03:34.174 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:03:34.174 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:03:34.174 INFO:tasks.workunit.client.0.trial112.stderr:134873: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:03:34.174 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:03:34.174 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:03:34.174 INFO:tasks.workunit.client.0.trial112.stderr:134873: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:03:34.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-13T02:03:34.175 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-13T02:03:34.175 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 134872 2026-02-13T02:03:34.186 INFO:tasks.workunit.client.0.trial112.stderr:ers.sh:2267: flush_pg_stats: echo 3-339302416386 2026-02-13T02:03:34.186 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:34.187 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:03:34.187 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-339302416386 2026-02-13T02:03:34.187 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:34.187 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=339302416386 2026-02-13T02:03:34.187 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 339302416386' 2026-02-13T02:03:34.187 INFO:tasks.workunit.client.0.trial112.stderr:134875: waiting osd.3 seq 339302416386 2026-02-13T02:03:34.187 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:03:34.187 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 339302416387 -lt 339302416386 2026-02-13T02:03:34.187 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:03:34.187 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:03:34.187 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:03:34.187 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:03:34.188 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:03:34.188 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:03:34.188 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:03:34.188 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:03:34.188 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:03:34.188 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:03:34.188 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:03:34.188 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:03:34.188 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:03:34.188 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:03:34.188 INFO:tasks.workunit.client.0.trial112.stderr:134875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:03:34.188 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:03:34.188 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:03:34.189 INFO:tasks.workunit.client.0.trial112.stderr:134875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:03:34.190 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-13T02:03:34.190 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-13T02:03:34.190 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-13T02:03:34.190 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-13T02:03:34.190 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:272: corrupt_and_repair_two: return_code=0 2026-02-13T02:03:34.190 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-13T02:03:34.190 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:275: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-13T02:03:34.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:276: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-13T02:03:34.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-13T02:03:34.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:03:34.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:03:34.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:03:34.208 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:03:34.208 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:03:34.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:03:34.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:03:34.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:03:34.319 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:03:34.320 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:03:34.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:03:34.321 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:03:34.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:03:34.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:03:34.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:03:34.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:03:34.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:03:34.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:03:34.325 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:03:34.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:03:34.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:03:34.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:03:34.352 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:03:34.352 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:03:34.353 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:03:34.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:03:34.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:03:34.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:03:34.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-13T02:03:34.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-13T02:03:34.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-13T02:03:34.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-13T02:03:34.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:03:34.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:03:34.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:03:34.356 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:03:34.356 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:03:34.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:03:34.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:03:34.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:03:34.357 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:03:34.358 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:03:34.358 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:03:34.359 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:03:34.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:03:34.361 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:03:34.361 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:03:34.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:03:34.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:03:34.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:03:34.363 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:03:34.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:03:34.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:03:34.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:03:34.366 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:03:34.367 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:03:34.367 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:03:34.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:03:34.368 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:03:34.368 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:03:34.368 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-13T02:03:34.369 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T02:03:34.369 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:03:34.369 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:03:34.369 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.35207 2026-02-13T02:03:34.371 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T02:03:34.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-13T02:03:34.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T02:03:34.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-13T02:03:34.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_jerasure_overwrites td/osd-scrub-repair 2026-02-13T02:03:34.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:878: TEST_corrupt_and_repair_jerasure_overwrites: '[' true = true ']' 2026-02-13T02:03:34.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:879: TEST_corrupt_and_repair_jerasure_overwrites: corrupt_and_repair_jerasure td/osd-scrub-repair true 2026-02-13T02:03:34.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:857: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2026-02-13T02:03:34.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:858: corrupt_and_repair_jerasure: local allow_overwrites=true 2026-02-13T02:03:34.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: corrupt_and_repair_jerasure: local poolname=ecpool 2026-02-13T02:03:34.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:861: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2026-02-13T02:03:34.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-13T02:03:34.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-13T02:03:34.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-13T02:03:34.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-13T02:03:34.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-13T02:03:34.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-13T02:03:34.393 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-13T02:03:34.393 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:03:34.394 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:03:34.394 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:03:34.394 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:03:34.394 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:03:34.394 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:03:34.395 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-13T02:03:34.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-13T02:03:34.424 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-13T02:03:34.424 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:03:34.425 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:03:34.425 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-13T02:03:34.426 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-13T02:03:34.426 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:03:34.426 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:03:34.427 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:03:34.427 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:03:34.427 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:03:34.427 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:03:34.427 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:03:34.427 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:03:34.427 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get fsid 2026-02-13T02:03:34.484 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-13T02:03:34.484 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:03:34.484 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:03:34.485 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-13T02:03:34.485 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-13T02:03:34.486 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:03:34.486 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:03:34.486 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:03:34.486 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:03:34.486 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:03:34.487 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:03:34.487 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:03:34.487 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:03:34.487 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get mon_host 2026-02-13T02:03:34.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:862: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2026-02-13T02:03:34.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-13T02:03:34.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-13T02:03:34.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-13T02:03:34.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-13T02:03:34.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-13T02:03:34.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-13T02:03:34.640 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-13T02:03:34.640 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:03:34.640 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:03:34.641 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:03:34.641 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:03:34.641 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:03:34.641 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:03:34.642 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:03:34.643 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:03:34.660 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: corrupt_and_repair_jerasure: seq 0 3 2026-02-13T02:03:34.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2026-02-13T02:03:34.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:864: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2026-02-13T02:03:34.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:03:34.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:03:34.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-13T02:03:34.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:03:34.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:03:34.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:03:34.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:03:34.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:03:34.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:03:34.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:03:34.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:03:34.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:03:34.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:03:34.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:03:34.663 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:03:34.663 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:03:34.663 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:03:34.664 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:03:34.664 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:03:34.664 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:03:34.664 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:03:34.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:03:34.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:03:34.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:03:34.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:03:34.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:03:34.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:03:34.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:03:34.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:03:34.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:03:34.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:03:34.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:03:34.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:03:34.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:03:34.667 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:03:34.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=9b175cf4-0b8f-4526-aa64-f12e810a085f 2026-02-13T02:03:34.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 9b175cf4-0b8f-4526-aa64-f12e810a085f' 2026-02-13T02:03:34.667 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 9b175cf4-0b8f-4526-aa64-f12e810a085f 2026-02-13T02:03:34.667 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:03:34.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB2ho5pfB75JxAAPSzI4hxOX2gYSVU/tB58/w== 2026-02-13T02:03:34.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB2ho5pfB75JxAAPSzI4hxOX2gYSVU/tB58/w=="}' 2026-02-13T02:03:34.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 9b175cf4-0b8f-4526-aa64-f12e810a085f -i td/osd-scrub-repair/0/new.json 2026-02-13T02:03:34.766 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:03:34.774 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-13T02:03:34.775 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB2ho5pfB75JxAAPSzI4hxOX2gYSVU/tB58/w== --osd-uuid 9b175cf4-0b8f-4526-aa64-f12e810a085f 2026-02-13T02:03:34.790 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:34.780+0000 7f6674739a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:34.792 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:34.784+0000 7f6674739a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:34.793 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:34.784+0000 7f6674739a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:34.793 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:34.784+0000 7f6674739a40 -1 bdev(0x555ba8a24c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-13T02:03:34.793 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:34.784+0000 7f6674739a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-13T02:03:35.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-13T02:03:35.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:03:35.505 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-13T02:03:35.506 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T02:03:35.506 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:03:35.600 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T02:03:35.601 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-13T02:03:35.601 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:03:35.601 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:03:35.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:03:35.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:03:35.616 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:35.603+0000 7f02baebaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:35.617 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:35.607+0000 7f02baebaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:35.618 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:35.607+0000 7f02baebaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:35.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-13T02:03:35.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:03:35.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:03:35.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:03:35.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:03:35.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:03:35.698 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:03:35.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:03:35.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:03:35.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:03:35.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:03:36.042 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:36.031+0000 7f02baebaa40 -1 Falling back to public interface 2026-02-13T02:03:36.668 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:36.655+0000 7f02baebaa40 -1 osd.0 0 log_to_monitors true 2026-02-13T02:03:36.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:03:36.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:03:36.788 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:03:36.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:03:36.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:03:36.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:03:36.905 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:03:37.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:03:37.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:03:37.908 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:03:37.908 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:03:37.908 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:03:37.908 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:03:38.072 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2685555124,v1:127.0.0.1:6801/2685555124] [v2:127.0.0.1:6802/2685555124,v1:127.0.0.1:6803/2685555124] exists,up 9b175cf4-0b8f-4526-aa64-f12e810a085f 2026-02-13T02:03:38.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:03:38.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:03:38.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:03:38.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2026-02-13T02:03:38.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:864: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2026-02-13T02:03:38.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:03:38.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:03:38.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-13T02:03:38.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:03:38.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:03:38.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:03:38.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:03:38.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:03:38.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:03:38.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:03:38.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:03:38.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:03:38.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:03:38.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:03:38.077 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:03:38.077 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:03:38.077 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:03:38.077 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:03:38.077 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:03:38.077 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:03:38.077 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:03:38.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:03:38.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:03:38.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:03:38.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:03:38.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:03:38.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:03:38.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:03:38.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:03:38.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:03:38.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:03:38.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:03:38.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:03:38.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:03:38.078 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:03:38.079 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e2e6d346-242c-41aa-a5a5-fbdd15bada25 2026-02-13T02:03:38.079 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 e2e6d346-242c-41aa-a5a5-fbdd15bada25' 2026-02-13T02:03:38.080 INFO:tasks.workunit.client.0.trial112.stdout:add osd1 e2e6d346-242c-41aa-a5a5-fbdd15bada25 2026-02-13T02:03:38.080 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:03:38.089 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB6ho5p5ir1BBAA3ry8cT2C/2t9/wxkW6oQHQ== 2026-02-13T02:03:38.089 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB6ho5p5ir1BBAA3ry8cT2C/2t9/wxkW6oQHQ=="}' 2026-02-13T02:03:38.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e2e6d346-242c-41aa-a5a5-fbdd15bada25 -i td/osd-scrub-repair/1/new.json 2026-02-13T02:03:38.252 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:03:38.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-13T02:03:38.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB6ho5p5ir1BBAA3ry8cT2C/2t9/wxkW6oQHQ== --osd-uuid e2e6d346-242c-41aa-a5a5-fbdd15bada25 2026-02-13T02:03:38.279 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:38.267+0000 7f609b3a9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:38.280 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:38.267+0000 7f609b3a9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:38.281 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:38.271+0000 7f609b3a9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:38.282 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:38.271+0000 7f609b3a9a40 -1 bdev(0x5559dff18c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-13T02:03:38.282 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:38.271+0000 7f609b3a9a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-13T02:03:38.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-13T02:03:38.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:03:38.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-13T02:03:38.921 INFO:tasks.workunit.client.0.trial112.stdout:adding osd1 key to auth repository 2026-02-13T02:03:38.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:03:39.129 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T02:03:39.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-13T02:03:39.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:03:39.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:03:39.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:03:39.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:03:39.153 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:39.143+0000 7f5463ea5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:39.155 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:39.143+0000 7f5463ea5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:39.156 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:39.147+0000 7f5463ea5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:39.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-13T02:03:39.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:03:39.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:03:39.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:03:39.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:03:39.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:03:39.287 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:03:39.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:03:39.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:03:39.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:03:39.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:03:39.554 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:39.543+0000 7f5463ea5a40 -1 Falling back to public interface 2026-02-13T02:03:40.186 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:40.175+0000 7f5463ea5a40 -1 osd.1 0 log_to_monitors true 2026-02-13T02:03:40.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:03:40.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:03:40.455 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:03:40.455 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:03:40.455 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:03:40.455 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:03:40.620 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:03:41.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:03:41.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:03:41.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:03:41.622 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:03:41.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:03:41.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:03:41.785 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/142486081,v1:127.0.0.1:6811/142486081] [v2:127.0.0.1:6812/142486081,v1:127.0.0.1:6813/142486081] exists,up e2e6d346-242c-41aa-a5a5-fbdd15bada25 2026-02-13T02:03:41.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:03:41.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:03:41.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:03:41.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2026-02-13T02:03:41.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:864: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2026-02-13T02:03:41.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:03:41.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:03:41.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-13T02:03:41.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:03:41.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-13T02:03:41.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:03:41.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:03:41.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:03:41.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:03:41.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-13T02:03:41.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-13T02:03:41.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:03:41.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:03:41.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:03:41.787 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:03:41.787 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:03:41.787 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:03:41.787 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:03:41.787 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:03:41.787 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:03:41.787 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:03:41.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:03:41.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:03:41.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:03:41.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:03:41.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:03:41.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:03:41.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:03:41.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:03:41.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:03:41.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:03:41.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:03:41.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:03:41.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-13T02:03:41.789 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:03:41.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=28a899a3-d0e3-4eb8-b652-37d882ac97d0 2026-02-13T02:03:41.791 INFO:tasks.workunit.client.0.trial112.stdout:add osd2 28a899a3-d0e3-4eb8-b652-37d882ac97d0 2026-02-13T02:03:41.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 28a899a3-d0e3-4eb8-b652-37d882ac97d0' 2026-02-13T02:03:41.791 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:03:41.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB9ho5pq4R9LxAA+xnkiXEuBQHoiYXF4W8ppA== 2026-02-13T02:03:41.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB9ho5pq4R9LxAA+xnkiXEuBQHoiYXF4W8ppA=="}' 2026-02-13T02:03:41.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 28a899a3-d0e3-4eb8-b652-37d882ac97d0 -i td/osd-scrub-repair/2/new.json 2026-02-13T02:03:41.972 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:03:41.981 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-13T02:03:41.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB9ho5pq4R9LxAA+xnkiXEuBQHoiYXF4W8ppA== --osd-uuid 28a899a3-d0e3-4eb8-b652-37d882ac97d0 2026-02-13T02:03:41.998 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:41.987+0000 7f23c557aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:42.000 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:41.987+0000 7f23c557aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:42.001 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:41.991+0000 7f23c557aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:42.001 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:41.991+0000 7f23c557aa40 -1 bdev(0x5625b55e6c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-13T02:03:42.002 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:41.991+0000 7f23c557aa40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-13T02:03:42.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-13T02:03:42.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:03:42.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-13T02:03:42.605 INFO:tasks.workunit.client.0.trial112.stdout:adding osd2 key to auth repository 2026-02-13T02:03:42.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:03:42.812 INFO:tasks.workunit.client.0.trial112.stdout:start osd.2 2026-02-13T02:03:42.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-13T02:03:42.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:03:42.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:03:42.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:03:42.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:03:42.836 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:42.827+0000 7fe7abb60a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:42.838 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:42.827+0000 7fe7abb60a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:42.839 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:42.827+0000 7fe7abb60a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:42.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-13T02:03:42.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:03:42.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T02:03:42.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:03:42.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:03:42.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:03:42.973 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:03:42.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:03:42.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:03:42.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:03:43.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:03:43.262 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:43.251+0000 7fe7abb60a40 -1 Falling back to public interface 2026-02-13T02:03:43.911 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:43.899+0000 7fe7abb60a40 -1 osd.2 0 log_to_monitors true 2026-02-13T02:03:44.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:03:44.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:03:44.141 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:03:44.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:03:44.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:03:44.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:03:44.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:03:45.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:03:45.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:03:45.308 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:03:45.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:03:45.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:03:45.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:03:45.469 INFO:tasks.workunit.client.0.trial112.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2693792877,v1:127.0.0.1:6819/2693792877] [v2:127.0.0.1:6820/2693792877,v1:127.0.0.1:6821/2693792877] exists,up 28a899a3-d0e3-4eb8-b652-37d882ac97d0 2026-02-13T02:03:45.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:03:45.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:03:45.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:03:45.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2026-02-13T02:03:45.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:864: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2026-02-13T02:03:45.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:03:45.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:03:45.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-13T02:03:45.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:03:45.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:03:45.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:03:45.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:03:45.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:03:45.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:03:45.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-13T02:03:45.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-13T02:03:45.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:03:45.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:03:45.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:03:45.474 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:03:45.474 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:03:45.474 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:03:45.474 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:03:45.474 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:03:45.474 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:03:45.474 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:03:45.474 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:03:45.474 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:03:45.474 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:03:45.474 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:03:45.474 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:03:45.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:03:45.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:03:45.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:03:45.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:03:45.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:03:45.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:03:45.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:03:45.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/3 2026-02-13T02:03:45.476 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:03:45.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4ffbcaae-5aa3-42b7-b511-2f4ca69d4da7 2026-02-13T02:03:45.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 4ffbcaae-5aa3-42b7-b511-2f4ca69d4da7' 2026-02-13T02:03:45.476 INFO:tasks.workunit.client.0.trial112.stdout:add osd3 4ffbcaae-5aa3-42b7-b511-2f4ca69d4da7 2026-02-13T02:03:45.477 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:03:45.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCBho5p1gCnHBAAUrD9gSQZqPNd4XfO/SJFGw== 2026-02-13T02:03:45.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCBho5p1gCnHBAAUrD9gSQZqPNd4XfO/SJFGw=="}' 2026-02-13T02:03:45.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4ffbcaae-5aa3-42b7-b511-2f4ca69d4da7 -i td/osd-scrub-repair/3/new.json 2026-02-13T02:03:45.653 INFO:tasks.workunit.client.0.trial112.stdout:3 2026-02-13T02:03:45.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/3/new.json 2026-02-13T02:03:45.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCBho5p1gCnHBAAUrD9gSQZqPNd4XfO/SJFGw== --osd-uuid 4ffbcaae-5aa3-42b7-b511-2f4ca69d4da7 2026-02-13T02:03:45.681 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:45.671+0000 7fd44c50aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:45.683 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:45.671+0000 7fd44c50aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:45.684 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:45.671+0000 7fd44c50aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:45.684 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:45.675+0000 7fd44c50aa40 -1 bdev(0x55ee0c8f2c00 td/osd-scrub-repair/3/block) open stat got: (1) Operation not permitted 2026-02-13T02:03:45.684 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:45.675+0000 7fd44c50aa40 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2026-02-13T02:03:46.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2026-02-13T02:03:46.295 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:03:46.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-13T02:03:46.296 INFO:tasks.workunit.client.0.trial112.stdout:adding osd3 key to auth repository 2026-02-13T02:03:46.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:03:46.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-13T02:03:46.500 INFO:tasks.workunit.client.0.trial112.stdout:start osd.3 2026-02-13T02:03:46.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:03:46.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:03:46.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:03:46.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:03:46.524 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:46.511+0000 7f40ec280a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:46.526 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:46.515+0000 7f40ec280a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:46.527 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:46.515+0000 7f40ec280a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:03:46.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-13T02:03:46.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:03:46.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-13T02:03:46.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:03:46.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:03:46.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:03:46.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:03:46.662 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:03:46.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:03:46.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:03:46.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:03:46.906 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:46.895+0000 7f40ec280a40 -1 Falling back to public interface 2026-02-13T02:03:47.506 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:03:47.495+0000 7f40ec280a40 -1 osd.3 0 log_to_monitors true 2026-02-13T02:03:47.827 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:03:47.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:03:47.828 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:03:47.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:03:47.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:03:47.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:03:47.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:03:48.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:03:48.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:03:48.986 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:03:48.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:03:48.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:03:48.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:03:49.146 INFO:tasks.workunit.client.0.trial112.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2206574826,v1:127.0.0.1:6827/2206574826] [v2:127.0.0.1:6828/2206574826,v1:127.0.0.1:6829/2206574826] exists,up 4ffbcaae-5aa3-42b7-b511-2f4ca69d4da7 2026-02-13T02:03:49.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:03:49.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:03:49.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:03:49.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:866: corrupt_and_repair_jerasure: create_rbd_pool 2026-02-13T02:03:49.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-13T02:03:49.302 INFO:tasks.workunit.client.0.trial112.stderr:pool 'rbd' does not exist 2026-02-13T02:03:49.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-13T02:03:49.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-13T02:03:49.539 INFO:tasks.workunit.client.0.trial112.stderr:pool 'rbd' created 2026-02-13T02:03:49.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:03:50.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-13T02:03:50.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:867: corrupt_and_repair_jerasure: wait_for_clean 2026-02-13T02:03:50.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:03:50.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:03:50.869 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:03:50.869 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:03:50.869 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:03:50.869 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:03:50.870 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:03:50.870 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:03:50.870 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:03:50.981 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:03:50.981 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:03:50.981 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:03:50.981 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:03:50.981 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:03:50.982 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:03:51.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:03:51.141 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:03:51.141 INFO:tasks.workunit.client.0.trial112.stderr:2 2026-02-13T02:03:51.141 INFO:tasks.workunit.client.0.trial112.stderr:3' 2026-02-13T02:03:51.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:03:51.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:51.141 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:03:51.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836484 2026-02-13T02:03:51.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836484 2026-02-13T02:03:51.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484' 2026-02-13T02:03:51.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:51.216 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:03:51.289 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672963 2026-02-13T02:03:51.290 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672963 2026-02-13T02:03:51.290 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-42949672963' 2026-02-13T02:03:51.290 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:51.290 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:03:51.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509443 2026-02-13T02:03:51.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509443 2026-02-13T02:03:51.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-42949672963 2-64424509443' 2026-02-13T02:03:51.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:51.366 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:03:51.439 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=81604378626 2026-02-13T02:03:51.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 81604378626 2026-02-13T02:03:51.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-42949672963 2-64424509443 3-81604378626' 2026-02-13T02:03:51.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:51.440 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836484 2026-02-13T02:03:51.440 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:51.441 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:03:51.442 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836484 2026-02-13T02:03:51.442 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:51.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836484 2026-02-13T02:03:51.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-13T02:03:51.444 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836484 2026-02-13T02:03:51.445 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:03:51.610 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836484 2026-02-13T02:03:51.611 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:03:52.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:03:52.612 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:03:52.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836484 2026-02-13T02:03:52.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:03:53.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:03:53.778 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:03:53.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836484 2026-02-13T02:03:53.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:53.940 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672963 2026-02-13T02:03:53.940 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:53.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:03:53.942 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672963 2026-02-13T02:03:53.942 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:53.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672963 2026-02-13T02:03:53.944 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 42949672963 2026-02-13T02:03:53.944 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2026-02-13T02:03:53.944 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:03:54.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672964 -lt 42949672963 2026-02-13T02:03:54.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:54.117 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509443 2026-02-13T02:03:54.117 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:54.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:03:54.119 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509443 2026-02-13T02:03:54.119 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:54.121 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509443 2026-02-13T02:03:54.121 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 64424509443 2026-02-13T02:03:54.121 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509443' 2026-02-13T02:03:54.121 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:03:54.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509443 -lt 64424509443 2026-02-13T02:03:54.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:54.282 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-81604378626 2026-02-13T02:03:54.282 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:54.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:03:54.284 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-81604378626 2026-02-13T02:03:54.284 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:54.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=81604378626 2026-02-13T02:03:54.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 81604378626' 2026-02-13T02:03:54.286 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.3 seq 81604378626 2026-02-13T02:03:54.286 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:03:54.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 81604378626 -lt 81604378626 2026-02-13T02:03:54.451 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:03:54.451 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:03:54.451 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:03:54.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-13T02:03:54.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:03:54.647 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:03:54.647 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:03:54.647 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:03:54.647 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:03:54.647 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:03:54.647 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:03:54.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-13T02:03:54.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:03:54.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:03:54.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:03:55.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-13T02:03:55.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:03:55.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:03:55.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:869: corrupt_and_repair_jerasure: create_ec_pool ecpool true k=2 m=2 2026-02-13T02:03:55.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2522: create_ec_pool: local pool_name=ecpool 2026-02-13T02:03:55.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2523: create_ec_pool: shift 2026-02-13T02:03:55.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2524: create_ec_pool: local allow_overwrites=true 2026-02-13T02:03:55.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2525: create_ec_pool: shift 2026-02-13T02:03:55.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2527: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2026-02-13T02:03:55.181 INFO:tasks.workunit.client.0.trial112.stderr:load: isa 2026-02-13T02:03:55.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2529: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2026-02-13T02:03:55.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2026-02-13T02:03:55.485 INFO:tasks.workunit.client.0.trial112.stderr:pool 'ecpool' created 2026-02-13T02:03:55.497 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:03:56.498 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2531: create_ec_pool: '[' true = true ']' 2026-02-13T02:03:56.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2532: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2026-02-13T02:03:56.705 INFO:tasks.workunit.client.0.trial112.stderr:set pool 2 allow_ec_overwrites to true 2026-02-13T02:03:56.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2535: create_ec_pool: wait_for_clean 2026-02-13T02:03:56.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:03:56.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:03:56.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:03:56.717 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:03:56.717 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:03:56.717 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:03:56.717 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:03:56.717 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:03:56.717 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:03:56.836 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:03:56.836 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:03:56.836 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:03:56.836 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:03:56.836 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:03:56.836 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:03:57.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:03:57.003 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:03:57.003 INFO:tasks.workunit.client.0.trial112.stderr:2 2026-02-13T02:03:57.003 INFO:tasks.workunit.client.0.trial112.stderr:3' 2026-02-13T02:03:57.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:03:57.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:57.003 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:03:57.079 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836486 2026-02-13T02:03:57.079 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836486 2026-02-13T02:03:57.079 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486' 2026-02-13T02:03:57.079 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:57.079 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:03:57.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672966 2026-02-13T02:03:57.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672966 2026-02-13T02:03:57.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486 1-42949672966' 2026-02-13T02:03:57.152 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:57.152 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:03:57.224 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509445 2026-02-13T02:03:57.224 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509445 2026-02-13T02:03:57.224 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486 1-42949672966 2-64424509445' 2026-02-13T02:03:57.224 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:03:57.224 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:03:57.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=81604378628 2026-02-13T02:03:57.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 81604378628 2026-02-13T02:03:57.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486 1-42949672966 2-64424509445 3-81604378628' 2026-02-13T02:03:57.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:57.297 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836486 2026-02-13T02:03:57.297 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:57.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:03:57.299 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836486 2026-02-13T02:03:57.299 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:57.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836486 2026-02-13T02:03:57.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2026-02-13T02:03:57.301 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836486 2026-02-13T02:03:57.301 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:03:57.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 21474836486 2026-02-13T02:03:57.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:03:58.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:03:58.477 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:03:58.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 21474836486 2026-02-13T02:03:58.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:03:59.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:03:59.641 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:03:59.806 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836486 2026-02-13T02:03:59.806 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:59.807 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672966 2026-02-13T02:03:59.807 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:59.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:03:59.809 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672966 2026-02-13T02:03:59.809 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:59.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672966 2026-02-13T02:03:59.811 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 42949672966 2026-02-13T02:03:59.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2026-02-13T02:03:59.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:03:59.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672966 -lt 42949672966 2026-02-13T02:03:59.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:03:59.979 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509445 2026-02-13T02:03:59.979 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:03:59.980 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:03:59.981 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509445 2026-02-13T02:03:59.981 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:03:59.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509445 2026-02-13T02:03:59.983 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 64424509445 2026-02-13T02:03:59.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2026-02-13T02:03:59.983 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:04:00.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509445 -lt 64424509445 2026-02-13T02:04:00.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:00.157 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-81604378628 2026-02-13T02:04:00.157 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:00.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:04:00.159 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-81604378628 2026-02-13T02:04:00.159 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:04:00.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=81604378628 2026-02-13T02:04:00.161 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.3 seq 81604378628 2026-02-13T02:04:00.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 81604378628' 2026-02-13T02:04:00.161 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:04:00.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 81604378629 -lt 81604378628 2026-02-13T02:04:00.325 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:04:00.326 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:04:00.326 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:04:00.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:04:00.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:04:00.700 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:04:00.700 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:04:00.701 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:04:00.701 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:04:00.701 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:04:00.701 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:04:00.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:04:00.701 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:04:00.701 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:04:00.701 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:04:00.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:04:00.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:04:00.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:04:00.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2536: create_ec_pool: return 0 2026-02-13T02:04:00.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:870: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2026-02-13T02:04:00.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:308: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2026-02-13T02:04:00.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:309: corrupt_and_repair_erasure_coded: local poolname=ecpool 2026-02-13T02:04:00.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:311: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2026-02-13T02:04:00.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:04:00.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-13T02:04:00.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2026-02-13T02:04:00.886 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:04:00.886 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:04:00.886 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:04:01.126 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:04:01.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:04:01.327 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:04:01.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:04:01.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:04:01.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-13T02:04:01.366 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:313: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2026-02-13T02:04:01.367 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2026-02-13T02:04:01.367 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-13T02:04:01.367 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2026-02-13T02:04:01.367 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T02:04:01.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:313: corrupt_and_repair_erasure_coded: local primary=3 2026-02-13T02:04:01.531 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2026-02-13T02:04:01.531 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: sed -e s/3// 2026-02-13T02:04:01.531 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2026-02-13T02:04:01.531 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-13T02:04:01.532 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2026-02-13T02:04:01.532 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-13T02:04:01.697 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-13T02:04:01.697 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:04:01.697 INFO:tasks.workunit.client.0.trial112.stderr:2 2026-02-13T02:04:01.698 INFO:tasks.workunit.client.0.trial112.stderr:0' 2026-02-13T02:04:01.698 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 0 2026-02-13T02:04:01.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: osds=('1' '2' '0') 2026-02-13T02:04:01.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: local -a osds 2026-02-13T02:04:01.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:315: corrupt_and_repair_erasure_coded: local not_primary_first=1 2026-02-13T02:04:01.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:316: corrupt_and_repair_erasure_coded: local not_primary_second=2 2026-02-13T02:04:01.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:319: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2026-02-13T02:04:01.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:286: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2026-02-13T02:04:01.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: corrupt_and_repair_one: local poolname=ecpool 2026-02-13T02:04:01.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:288: corrupt_and_repair_one: local osd=3 2026-02-13T02:04:01.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:293: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-13T02:04:01.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:04:01.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:04:01.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-13T02:04:01.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:04:01.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2026-02-13T02:04:01.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:04:01.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:04:01.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-13T02:04:01.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:04:01.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-13T02:04:01.702 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:04:01.702 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:04:01.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:04:01.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:04:01.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:04:02.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:04:02.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2026-02-13T02:04:02.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:04:02.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:04:02.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-13T02:04:02.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:04:02.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:04:02.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2026-02-13T02:04:02.213 INFO:tasks.workunit.client.0.trial112.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2026-02-13T02:04:02.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-13T02:04:02.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:04:02.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:04:02.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-13T02:04:02.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:04:02.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:04:02.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:04:02.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:04:02.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:04:02.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:04:02.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-13T02:04:02.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-13T02:04:02.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:04:02.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:04:02.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:04:02.304 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:04:02.304 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:04:02.304 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:04:02.304 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:04:02.304 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:04:02.304 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:04:02.304 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:04:02.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:04:02.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:04:02.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:04:02.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:04:02.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:04:02.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:04:02.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:04:02.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:04:02.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:04:02.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:04:02.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-13T02:04:02.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-13T02:04:02.307 INFO:tasks.workunit.client.0.trial112.stderr:start osd.3 2026-02-13T02:04:02.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:04:02.308 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-13T02:04:02.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-13T02:04:02.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:04:02.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:04:02.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:04:02.324 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:04:02.311+0000 7f5a9354da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:04:02.325 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:04:02.315+0000 7f5a9354da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:04:02.326 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:04:02.315+0000 7f5a9354da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:04:02.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-13T02:04:02.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:04:02.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-13T02:04:02.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:04:02.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:04:02.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:04:02.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:04:02.468 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:04:02.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:04:02.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:04:02.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:04:02.718 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:04:02.707+0000 7f5a9354da40 -1 Falling back to public interface 2026-02-13T02:04:03.346 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:04:03.335+0000 7f5a9354da40 -1 osd.3 35 log_to_monitors true 2026-02-13T02:04:03.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:04:03.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:04:03.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:04:03.632 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:04:03.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:04:03.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:04:03.792 INFO:tasks.workunit.client.0.trial112.stderr:osd.3 up in weight 1 up_from 39 up_thru 39 down_at 36 last_clean_interval [19,35) [v2:127.0.0.1:6826/2018240908,v1:127.0.0.1:6827/2018240908] [v2:127.0.0.1:6828/2018240908,v1:127.0.0.1:6829/2018240908] exists,up 4ffbcaae-5aa3-42b7-b511-2f4ca69d4da7 2026-02-13T02:04:03.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:04:03.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:04:03.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:04:03.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:04:03.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:04:03.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:04:03.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:04:03.793 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:04:03.794 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:04:03.794 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:04:03.795 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:04:03.795 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:04:03.795 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:04:03.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:04:03.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:04:03.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:04:03.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:04:03.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:04:03.907 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:04:04.072 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:04:04.073 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:04:04.073 INFO:tasks.workunit.client.0.trial112.stderr:2 2026-02-13T02:04:04.073 INFO:tasks.workunit.client.0.trial112.stderr:3' 2026-02-13T02:04:04.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:04:04.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:04:04.073 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:04:04.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836489 2026-02-13T02:04:04.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836489 2026-02-13T02:04:04.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489' 2026-02-13T02:04:04.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:04:04.148 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:04:04.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672968 2026-02-13T02:04:04.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672968 2026-02-13T02:04:04.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-42949672968' 2026-02-13T02:04:04.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:04:04.220 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:04:04.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509447 2026-02-13T02:04:04.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509447 2026-02-13T02:04:04.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509447' 2026-02-13T02:04:04.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:04:04.297 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:04:04.368 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=167503724546 2026-02-13T02:04:04.369 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 167503724546 2026-02-13T02:04:04.369 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509447 3-167503724546' 2026-02-13T02:04:04.369 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:04.369 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836489 2026-02-13T02:04:04.370 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:04.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:04:04.371 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836489 2026-02-13T02:04:04.371 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:04:04.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836489 2026-02-13T02:04:04.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2026-02-13T02:04:04.373 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 21474836489 2026-02-13T02:04:04.373 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:04:04.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836489 2026-02-13T02:04:04.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:04:05.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:04:05.534 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:04:05.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836489 2026-02-13T02:04:05.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:05.698 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672968 2026-02-13T02:04:05.699 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:05.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:04:05.701 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672968 2026-02-13T02:04:05.701 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:04:05.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672968 2026-02-13T02:04:05.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2026-02-13T02:04:05.702 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 42949672968 2026-02-13T02:04:05.703 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:04:05.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672968 -lt 42949672968 2026-02-13T02:04:05.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:05.875 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509447 2026-02-13T02:04:05.875 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:05.876 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:04:05.876 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509447 2026-02-13T02:04:05.877 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:04:05.878 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509447 2026-02-13T02:04:05.878 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2026-02-13T02:04:05.878 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 64424509447 2026-02-13T02:04:05.878 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:04:06.049 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509447 -lt 64424509447 2026-02-13T02:04:06.049 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:06.050 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-167503724546 2026-02-13T02:04:06.050 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:06.051 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:04:06.052 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-167503724546 2026-02-13T02:04:06.052 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:04:06.053 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=167503724546 2026-02-13T02:04:06.054 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 167503724546' 2026-02-13T02:04:06.054 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.3 seq 167503724546 2026-02-13T02:04:06.054 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:04:06.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 167503724546 -lt 167503724546 2026-02-13T02:04:06.215 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:04:06.216 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:04:06.216 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:04:06.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:04:06.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:04:06.406 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:04:06.406 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:04:06.406 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:04:06.406 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:04:06.406 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:04:06.406 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:04:06.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:04:06.562 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:04:06.562 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:04:06.562 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:04:06.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:04:06.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:04:06.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:04:06.752 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: get_pg ecpool SOMETHING 2026-02-13T02:04:06.752 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-13T02:04:06.752 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-13T02:04:06.752 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-13T02:04:06.752 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T02:04:06.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: local pg=2.0 2026-02-13T02:04:06.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: corrupt_and_repair_one: repair 2.0 2026-02-13T02:04:06.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-13T02:04:06.909 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-13T02:04:06.910 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:04:06.910 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:04:06.910 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:04:06.910 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:04:07.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-13T02:03:55.478161+0000 2026-02-13T02:04:07.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-13T02:04:07.215 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-13T02:04:07.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-13T02:03:55.478161+0000 2026-02-13T02:04:07.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-13T02:04:07.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:03:55.478161+0000 2026-02-13T02:04:07.228 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:04:07.228 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:04:07.228 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:04:07.228 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:04:07.228 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:04:07.228 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:04:07.229 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:04:07.229 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:04:07.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:03:55.478161+0000 '>' 2026-02-13T02:03:55.478161+0000 2026-02-13T02:04:07.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:04:08.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:04:08.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:04:08.389 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:04:08.390 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:04:08.390 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:04:08.390 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:04:08.390 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:04:08.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:03:55.478161+0000 '>' 2026-02-13T02:03:55.478161+0000 2026-02-13T02:04:08.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:04:09.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:04:09.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:04:09.551 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:04:09.551 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:04:09.552 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:04:09.552 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:04:09.552 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:04:09.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:04:07.239471+0000 '>' 2026-02-13T02:03:55.478161+0000 2026-02-13T02:04:09.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:04:09.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-13T02:04:09.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:04:09.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:04:09.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-13T02:04:09.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:04:09.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-13T02:04:09.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:04:09.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:04:09.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-13T02:04:09.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:04:09.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-13T02:04:09.717 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:04:09.717 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:04:09.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:04:09.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:04:09.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:04:10.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:04:10.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-13T02:04:10.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:04:10.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:04:10.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-13T02:04:10.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:04:10.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:04:10.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2026-02-13T02:04:10.092 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::deep_scrub_2.0s0:head#, (61) No data available 2026-02-13T02:04:10.092 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2026-02-13T02:04:10.092 INFO:tasks.workunit.client.0.trial112.stdout:_ 2026-02-13T02:04:10.093 INFO:tasks.workunit.client.0.trial112.stdout:hinfo_key 2026-02-13T02:04:10.093 INFO:tasks.workunit.client.0.trial112.stdout:snapset 2026-02-13T02:04:10.195 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-13T02:04:10.196 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:04:10.196 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:04:10.196 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-13T02:04:10.196 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:04:10.196 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:04:10.196 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:04:10.196 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:04:10.196 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:04:10.196 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:04:10.196 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-13T02:04:10.196 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-13T02:04:10.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:04:10.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:04:10.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:04:10.197 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:04:10.197 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:04:10.197 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:04:10.197 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:04:10.197 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:04:10.197 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:04:10.197 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:04:10.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:04:10.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:04:10.198 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:04:10.198 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:04:10.198 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:04:10.198 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:04:10.198 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:04:10.198 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:04:10.198 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:04:10.198 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:04:10.198 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-13T02:04:10.199 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-13T02:04:10.199 INFO:tasks.workunit.client.0.trial112.stderr:start osd.3 2026-02-13T02:04:10.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:04:10.200 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-13T02:04:10.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-13T02:04:10.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:04:10.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:04:10.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:04:10.214 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:04:10.206+0000 7f55bc9a5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:04:10.215 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:04:10.206+0000 7f55bc9a5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:04:10.216 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:04:10.206+0000 7f55bc9a5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:04:10.361 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-13T02:04:10.361 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:04:10.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-13T02:04:10.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:04:10.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:04:10.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:04:10.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:04:10.362 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:04:10.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:04:10.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:04:10.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:04:10.610 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:04:10.602+0000 7f55bc9a5a40 -1 Falling back to public interface 2026-02-13T02:04:11.215 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:04:11.206+0000 7f55bc9a5a40 -1 osd.3 40 log_to_monitors true 2026-02-13T02:04:11.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:04:11.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:04:11.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:04:11.522 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:04:11.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:04:11.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:04:11.678 INFO:tasks.workunit.client.0.trial112.stderr:osd.3 up in weight 1 up_from 44 up_thru 44 down_at 41 last_clean_interval [39,40) [v2:127.0.0.1:6826/2466161859,v1:127.0.0.1:6827/2466161859] [v2:127.0.0.1:6828/2466161859,v1:127.0.0.1:6829/2466161859] exists,up 4ffbcaae-5aa3-42b7-b511-2f4ca69d4da7 2026-02-13T02:04:11.678 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:04:11.678 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:04:11.678 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:04:11.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:04:11.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:04:11.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:04:11.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:04:11.679 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:04:11.680 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:04:11.680 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:04:11.680 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:04:11.680 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:04:11.680 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:04:11.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:04:11.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:04:11.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:04:11.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:04:11.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:04:11.794 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:04:11.950 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:04:11.950 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:04:11.950 INFO:tasks.workunit.client.0.trial112.stderr:2 2026-02-13T02:04:11.950 INFO:tasks.workunit.client.0.trial112.stderr:3' 2026-02-13T02:04:11.950 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:04:11.950 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:04:11.950 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:04:12.024 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836491 2026-02-13T02:04:12.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836491 2026-02-13T02:04:12.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491' 2026-02-13T02:04:12.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:04:12.025 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:04:12.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672971 2026-02-13T02:04:12.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672971 2026-02-13T02:04:12.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-42949672971' 2026-02-13T02:04:12.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:04:12.096 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:04:12.170 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509450 2026-02-13T02:04:12.170 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509450 2026-02-13T02:04:12.170 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-42949672971 2-64424509450' 2026-02-13T02:04:12.170 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:04:12.170 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:04:12.243 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=188978561026 2026-02-13T02:04:12.243 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 188978561026 2026-02-13T02:04:12.243 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-42949672971 2-64424509450 3-188978561026' 2026-02-13T02:04:12.243 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:12.244 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836491 2026-02-13T02:04:12.244 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:12.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:04:12.246 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836491 2026-02-13T02:04:12.246 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:04:12.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836491 2026-02-13T02:04:12.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2026-02-13T02:04:12.247 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 21474836491 2026-02-13T02:04:12.248 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:04:12.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836490 -lt 21474836491 2026-02-13T02:04:12.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:04:13.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:04:13.421 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:04:13.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836491 2026-02-13T02:04:13.584 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:13.584 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672971 2026-02-13T02:04:13.584 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:13.585 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:04:13.586 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672971 2026-02-13T02:04:13.586 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:04:13.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672971 2026-02-13T02:04:13.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2026-02-13T02:04:13.588 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 42949672971 2026-02-13T02:04:13.588 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:04:13.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672971 -lt 42949672971 2026-02-13T02:04:13.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:13.762 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509450 2026-02-13T02:04:13.762 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:13.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:04:13.764 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509450 2026-02-13T02:04:13.764 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:04:13.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509450 2026-02-13T02:04:13.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2026-02-13T02:04:13.765 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 64424509450 2026-02-13T02:04:13.766 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:04:13.928 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509450 -lt 64424509450 2026-02-13T02:04:13.928 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:13.929 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-188978561026 2026-02-13T02:04:13.929 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:13.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:04:13.930 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-188978561026 2026-02-13T02:04:13.931 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:04:13.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=188978561026 2026-02-13T02:04:13.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 188978561026' 2026-02-13T02:04:13.932 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.3 seq 188978561026 2026-02-13T02:04:13.933 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:04:14.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 188978561026 -lt 188978561026 2026-02-13T02:04:14.100 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:04:14.100 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:04:14.101 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:04:14.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:04:14.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:04:14.292 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:04:14.292 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:04:14.292 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:04:14.292 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:04:14.293 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:04:14.293 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:04:14.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:04:14.447 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:04:14.448 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:04:14.448 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:04:14.643 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:04:14.643 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:04:14.643 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:04:14.643 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-13T02:04:14.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-13T02:04:14.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:321: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2026-02-13T02:04:14.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:286: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2026-02-13T02:04:14.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: corrupt_and_repair_one: local poolname=ecpool 2026-02-13T02:04:14.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:288: corrupt_and_repair_one: local osd=1 2026-02-13T02:04:14.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:293: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2026-02-13T02:04:14.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:04:14.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:04:14.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:04:14.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:04:14.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2026-02-13T02:04:14.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:04:14.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:04:14.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:04:14.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:04:14.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:04:14.666 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:04:14.666 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:04:14.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:04:14.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:04:14.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:04:14.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:04:14.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2026-02-13T02:04:14.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:04:14.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:04:14.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:04:14.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:04:14.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:04:14.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2026-02-13T02:04:15.182 INFO:tasks.workunit.client.0.trial112.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2026-02-13T02:04:15.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-13T02:04:15.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:04:15.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:04:15.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:04:15.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:04:15.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:04:15.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:04:15.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:04:15.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:04:15.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:04:15.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:04:15.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:04:15.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:04:15.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:04:15.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:04:15.283 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:04:15.283 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:04:15.283 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:04:15.283 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:04:15.283 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:04:15.283 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:04:15.285 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:04:15.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:04:15.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:04:15.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:04:15.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:04:15.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:04:15.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:04:15.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:04:15.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:04:15.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:04:15.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:04:15.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:04:15.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:04:15.285 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:04:15.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:04:15.286 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:04:15.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:04:15.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:04:15.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:04:15.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:04:15.302 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:04:15.294+0000 7f2f6c24da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:04:15.304 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:04:15.294+0000 7f2f6c24da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:04:15.305 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:04:15.298+0000 7f2f6c24da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:04:15.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:04:15.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:04:15.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:04:15.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:04:15.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:04:15.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:04:15.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:04:15.449 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:04:15.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:04:15.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:04:15.613 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:04:15.706 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:04:15.698+0000 7f2f6c24da40 -1 Falling back to public interface 2026-02-13T02:04:16.351 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:04:16.342+0000 7f2f6c24da40 -1 osd.1 46 log_to_monitors true 2026-02-13T02:04:16.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:04:16.615 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:04:16.615 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:04:16.615 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:04:16.615 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:04:16.615 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:04:16.779 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 50 up_thru 50 down_at 47 last_clean_interval [10,46) [v2:127.0.0.1:6810/4187716232,v1:127.0.0.1:6811/4187716232] [v2:127.0.0.1:6812/4187716232,v1:127.0.0.1:6813/4187716232] exists,up e2e6d346-242c-41aa-a5a5-fbdd15bada25 2026-02-13T02:04:16.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:04:16.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:04:16.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:04:16.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:04:16.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:04:16.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:04:16.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:04:16.780 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:04:16.780 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:04:16.780 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:04:16.781 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:04:16.781 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:04:16.781 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:04:16.893 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:04:16.893 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:04:16.893 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:04:16.893 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:04:16.893 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:04:16.894 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:04:17.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:04:17.059 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:04:17.060 INFO:tasks.workunit.client.0.trial112.stderr:2 2026-02-13T02:04:17.060 INFO:tasks.workunit.client.0.trial112.stderr:3' 2026-02-13T02:04:17.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:04:17.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:04:17.060 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:04:17.132 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836493 2026-02-13T02:04:17.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836493 2026-02-13T02:04:17.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493' 2026-02-13T02:04:17.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:04:17.133 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:04:17.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=214748364802 2026-02-13T02:04:17.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 214748364802 2026-02-13T02:04:17.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-214748364802' 2026-02-13T02:04:17.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:04:17.207 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:04:17.280 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509452 2026-02-13T02:04:17.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509452 2026-02-13T02:04:17.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-214748364802 2-64424509452' 2026-02-13T02:04:17.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:04:17.281 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:04:17.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=188978561028 2026-02-13T02:04:17.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 188978561028 2026-02-13T02:04:17.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-214748364802 2-64424509452 3-188978561028' 2026-02-13T02:04:17.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:17.353 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836493 2026-02-13T02:04:17.353 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:17.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:04:17.354 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836493 2026-02-13T02:04:17.355 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:04:17.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836493 2026-02-13T02:04:17.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2026-02-13T02:04:17.356 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 21474836493 2026-02-13T02:04:17.357 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:04:17.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836492 -lt 21474836493 2026-02-13T02:04:17.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:04:18.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:04:18.534 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:04:18.694 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836492 -lt 21474836493 2026-02-13T02:04:18.694 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:04:19.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:04:19.696 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:04:19.857 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836494 -lt 21474836493 2026-02-13T02:04:19.857 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:19.858 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-214748364802 2026-02-13T02:04:19.858 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:19.859 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:04:19.860 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-214748364802 2026-02-13T02:04:19.860 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:04:19.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=214748364802 2026-02-13T02:04:19.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 214748364802' 2026-02-13T02:04:19.862 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 214748364802 2026-02-13T02:04:19.862 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:04:20.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 214748364802 -lt 214748364802 2026-02-13T02:04:20.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:20.025 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509452 2026-02-13T02:04:20.025 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:20.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:04:20.028 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509452 2026-02-13T02:04:20.028 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:04:20.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509452 2026-02-13T02:04:20.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2026-02-13T02:04:20.029 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 64424509452 2026-02-13T02:04:20.030 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:04:20.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509452 -lt 64424509452 2026-02-13T02:04:20.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:20.192 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-188978561028 2026-02-13T02:04:20.193 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:20.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:04:20.194 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-188978561028 2026-02-13T02:04:20.194 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:04:20.196 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=188978561028 2026-02-13T02:04:20.196 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 188978561028' 2026-02-13T02:04:20.196 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.3 seq 188978561028 2026-02-13T02:04:20.197 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:04:20.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 188978561028 -lt 188978561028 2026-02-13T02:04:20.355 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:04:20.355 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:04:20.355 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:04:20.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:04:20.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:04:20.544 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:04:20.544 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:04:20.544 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:04:20.544 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:04:20.544 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:04:20.544 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:04:20.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:04:20.703 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:04:20.703 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:04:20.704 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:04:20.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:04:20.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:04:20.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:04:20.895 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: get_pg ecpool SOMETHING 2026-02-13T02:04:20.895 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-13T02:04:20.895 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-13T02:04:20.895 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-13T02:04:20.896 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T02:04:21.053 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: local pg=2.0 2026-02-13T02:04:21.054 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: corrupt_and_repair_one: repair 2.0 2026-02-13T02:04:21.054 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-13T02:04:21.054 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-13T02:04:21.054 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:04:21.054 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:04:21.054 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:04:21.054 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:04:21.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-13T02:04:07.239471+0000 2026-02-13T02:04:21.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-13T02:04:21.369 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-13T02:04:21.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-13T02:04:07.239471+0000 2026-02-13T02:04:21.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-13T02:04:21.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:04:07.239471+0000 2026-02-13T02:04:21.381 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:04:21.381 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:04:21.381 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:04:21.381 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:04:21.381 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:04:21.381 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:04:21.381 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:04:21.381 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:04:21.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:04:07.239471+0000 '>' 2026-02-13T02:04:07.239471+0000 2026-02-13T02:04:21.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:04:22.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:04:22.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:04:22.540 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:04:22.541 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:04:22.541 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:04:22.541 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:04:22.541 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:04:22.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:04:07.239471+0000 '>' 2026-02-13T02:04:07.239471+0000 2026-02-13T02:04:22.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:04:23.704 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:04:23.705 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:04:23.706 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:04:23.706 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:04:23.706 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:04:23.706 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:04:23.706 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:04:23.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:04:07.239471+0000 '>' 2026-02-13T02:04:07.239471+0000 2026-02-13T02:04:23.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:04:24.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:04:24.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:04:24.869 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:04:24.869 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:04:24.870 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:04:24.870 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:04:24.870 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:04:25.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:04:07.239471+0000 '>' 2026-02-13T02:04:07.239471+0000 2026-02-13T02:04:25.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:04:26.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:04:26.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:04:26.028 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:04:26.028 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:04:26.029 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:04:26.029 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:04:26.029 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:04:26.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:04:07.239471+0000 '>' 2026-02-13T02:04:07.239471+0000 2026-02-13T02:04:26.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:04:27.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:04:27.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:04:27.193 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:04:27.193 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:04:27.193 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:04:27.194 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:04:27.194 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:04:27.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:04:22.198682+0000 '>' 2026-02-13T02:04:07.239471+0000 2026-02-13T02:04:27.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:04:27.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-13T02:04:27.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:04:27.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:04:27.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-13T02:04:27.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:04:27.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-13T02:04:27.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:04:27.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:04:27.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-13T02:04:27.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:04:27.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-13T02:04:27.356 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:04:27.356 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:04:27.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:04:27.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:04:27.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:04:27.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:04:27.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-13T02:04:27.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:04:27.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:04:27.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-13T02:04:27.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:04:27.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:04:27.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2026-02-13T02:04:27.740 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::deep_scrub_2.0s0:head#, (61) No data available 2026-02-13T02:04:27.741 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2026-02-13T02:04:27.741 INFO:tasks.workunit.client.0.trial112.stdout:_ 2026-02-13T02:04:27.741 INFO:tasks.workunit.client.0.trial112.stdout:hinfo_key 2026-02-13T02:04:27.741 INFO:tasks.workunit.client.0.trial112.stdout:snapset 2026-02-13T02:04:27.839 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-13T02:04:27.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:04:27.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:04:27.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-13T02:04:27.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:04:27.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:04:27.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:04:27.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:04:27.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:04:27.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:04:27.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-13T02:04:27.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-13T02:04:27.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:04:27.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:04:27.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:04:27.841 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:04:27.841 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:04:27.841 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:04:27.841 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:04:27.841 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:04:27.841 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:04:27.841 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:04:27.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:04:27.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:04:27.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:04:27.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:04:27.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:04:27.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:04:27.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:04:27.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:04:27.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:04:27.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:04:27.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-13T02:04:27.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-13T02:04:27.842 INFO:tasks.workunit.client.0.trial112.stderr:start osd.3 2026-02-13T02:04:27.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:04:27.843 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-13T02:04:27.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-13T02:04:27.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:04:27.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:04:27.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:04:27.858 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:04:27.850+0000 7f8d10de5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:04:27.860 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:04:27.850+0000 7f8d10de5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:04:27.861 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:04:27.854+0000 7f8d10de5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:04:28.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-13T02:04:28.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:04:28.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-13T02:04:28.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:04:28.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:04:28.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:04:28.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:04:28.008 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:04:28.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:04:28.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:04:28.170 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:04:28.254 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:04:28.246+0000 7f8d10de5a40 -1 Falling back to public interface 2026-02-13T02:04:28.876 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:04:28.866+0000 7f8d10de5a40 -1 osd.3 52 log_to_monitors true 2026-02-13T02:04:29.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:04:29.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:04:29.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:04:29.172 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:04:29.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:04:29.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:04:29.333 INFO:tasks.workunit.client.0.trial112.stderr:osd.3 up in weight 1 up_from 56 up_thru 56 down_at 53 last_clean_interval [44,52) [v2:127.0.0.1:6826/1483294991,v1:127.0.0.1:6827/1483294991] [v2:127.0.0.1:6828/1483294991,v1:127.0.0.1:6829/1483294991] exists,up 4ffbcaae-5aa3-42b7-b511-2f4ca69d4da7 2026-02-13T02:04:29.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:04:29.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:04:29.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:04:29.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:04:29.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:04:29.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:04:29.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:04:29.334 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:04:29.335 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:04:29.335 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:04:29.335 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:04:29.335 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:04:29.335 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:04:29.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:04:29.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:04:29.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:04:29.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:04:29.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:04:29.445 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:04:29.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:04:29.609 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:04:29.609 INFO:tasks.workunit.client.0.trial112.stderr:2 2026-02-13T02:04:29.609 INFO:tasks.workunit.client.0.trial112.stderr:3' 2026-02-13T02:04:29.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:04:29.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:04:29.609 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:04:29.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836497 2026-02-13T02:04:29.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836497 2026-02-13T02:04:29.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497' 2026-02-13T02:04:29.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:04:29.687 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:04:29.760 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=214748364805 2026-02-13T02:04:29.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 214748364805 2026-02-13T02:04:29.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-214748364805' 2026-02-13T02:04:29.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:04:29.761 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:04:29.836 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509455 2026-02-13T02:04:29.836 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509455 2026-02-13T02:04:29.836 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-214748364805 2-64424509455' 2026-02-13T02:04:29.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:04:29.837 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:04:29.906 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=240518168578 2026-02-13T02:04:29.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 240518168578 2026-02-13T02:04:29.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-214748364805 2-64424509455 3-240518168578' 2026-02-13T02:04:29.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:29.907 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836497 2026-02-13T02:04:29.907 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:29.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:04:29.910 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836497 2026-02-13T02:04:29.910 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:04:29.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836497 2026-02-13T02:04:29.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2026-02-13T02:04:29.913 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 21474836497 2026-02-13T02:04:29.913 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:04:30.086 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836496 -lt 21474836497 2026-02-13T02:04:30.086 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:04:31.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:04:31.088 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:04:31.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836497 -lt 21474836497 2026-02-13T02:04:31.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:31.249 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-214748364805 2026-02-13T02:04:31.249 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:31.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:04:31.251 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-214748364805 2026-02-13T02:04:31.251 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:04:31.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=214748364805 2026-02-13T02:04:31.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 214748364805' 2026-02-13T02:04:31.253 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 214748364805 2026-02-13T02:04:31.253 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:04:31.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 214748364805 -lt 214748364805 2026-02-13T02:04:31.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:31.427 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509455 2026-02-13T02:04:31.428 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:31.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:04:31.429 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509455 2026-02-13T02:04:31.430 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:04:31.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509455 2026-02-13T02:04:31.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2026-02-13T02:04:31.431 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 64424509455 2026-02-13T02:04:31.431 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:04:31.594 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509456 -lt 64424509455 2026-02-13T02:04:31.594 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:31.595 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-240518168578 2026-02-13T02:04:31.595 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:31.596 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:04:31.597 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-240518168578 2026-02-13T02:04:31.598 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:04:31.599 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=240518168578 2026-02-13T02:04:31.599 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 240518168578' 2026-02-13T02:04:31.599 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.3 seq 240518168578 2026-02-13T02:04:31.600 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:04:31.766 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 240518168578 -lt 240518168578 2026-02-13T02:04:31.767 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:04:31.767 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:04:31.767 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:04:31.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:04:31.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:04:31.968 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:04:31.968 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:04:31.968 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:04:31.968 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:04:31.968 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:04:31.968 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:04:32.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:04:32.132 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:04:32.132 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:04:32.132 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:04:32.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:04:32.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:04:32.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:04:32.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-13T02:04:32.348 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-13T02:04:32.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:322: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2026-02-13T02:04:32.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:245: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2026-02-13T02:04:32.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:246: corrupt_and_repair_two: local poolname=ecpool 2026-02-13T02:04:32.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:247: corrupt_and_repair_two: local first=1 2026-02-13T02:04:32.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_two: local second=2 2026-02-13T02:04:32.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_two: pids= 2026-02-13T02:04:32.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2026-02-13T02:04:32.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-13T02:04:32.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-13T02:04:32.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 149386"' 2026-02-13T02:04:32.351 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 149386' 2026-02-13T02:04:32.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2026-02-13T02:04:32.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-13T02:04:32.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-13T02:04:32.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 149388"' 2026-02-13T02:04:32.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2026-02-13T02:04:32.352 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 149388' 2026-02-13T02:04:32.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_two: wait_background pids 2026-02-13T02:04:32.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 149386 149388' 2026-02-13T02:04:32.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-13T02:04:32.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-13T02:04:32.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 149386 2026-02-13T02:04:32.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/149389: /' 2026-02-13T02:04:32.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2026-02-13T02:04:32.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/149392: /' 2026-02-13T02:04:32.957 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:04:32.957 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:04:32.957 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:04:32.957 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:04:32.957 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2026-02-13T02:04:32.957 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:04:32.957 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:04:32.957 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:04:32.957 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:04:32.957 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:04:32.957 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:04:32.957 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:04:32.957 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:04:32.957 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:04:32.957 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:04:32.957 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:04:32.957 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2026-02-13T02:04:32.957 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:04:32.957 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:04:32.957 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:04:32.957 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:04:32.957 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:04:32.957 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2026-02-13T02:04:32.957 INFO:tasks.workunit.client.0.trial112.stderr:149389: remove 1#2:eb822e21:::SOMETHING:head# 2026-02-13T02:04:32.957 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-13T02:04:32.957 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:04:32.957 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:04:32.957 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:04:32.957 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:04:32.958 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:04:32.958 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:04:32.958 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:04:32.960 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:04:32.960 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:04:32.960 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:04:32.960 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:04:32.960 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:04:32.960 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:04:32.960 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:04:32.960 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:04:32.960 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:04:32.960 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:04:32.960 INFO:tasks.workunit.client.0.trial112.stderr:149389: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:04:32.960 INFO:tasks.workunit.client.0.trial112.stderr:149389: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:04:32.960 INFO:tasks.workunit.client.0.trial112.stderr:149389: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:04:32.960 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:04:32.960 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:04:32.961 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:04:32.961 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:04:32.961 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:04:32.961 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:04:32.961 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:04:32.961 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:04:32.961 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:04:32.961 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:04:32.961 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:04:32.961 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:04:32.961 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:04:32.961 INFO:tasks.workunit.client.0.trial112.stderr:149389: start osd.1 2026-02-13T02:04:32.961 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:04:32.985 INFO:tasks.workunit.client.0.trial112.stderr:1149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:04:32.985 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:04:32.985 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-13T02:04:32.985 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:04:32.985 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2026-02-13T02:04:32.986 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:04:32.986 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:04:32.986 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-13T02:04:32.986 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:04:32.986 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2026-02-13T02:04:32.986 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:04:32.986 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:04:32.986 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:04:32.986 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:04:32.986 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:04:32.986 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:04:32.986 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2026-02-13T02:04:32.986 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:04:32.986 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:04:32.986 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-13T02:04:32.986 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:04:32.986 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2026-02-13T02:04:32.986 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2026-02-13T02:04:32.986 INFO:tasks.workunit.client.0.trial112.stderr:149392: remove 2#2:eb822e21:::SOMETHING:head# 2026-02-13T02:04:32.986 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2026-02-13T02:04:32.986 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:04:32.986 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:04:32.986 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-13T02:04:32.986 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:04:32.986 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2026-02-13T02:04:32.986 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:04:32.986 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:04:32.989 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:04:32.989 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:04:32.989 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-13T02:04:32.989 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-13T02:04:32.989 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:04:32.989 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:04:32.989 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:04:32.989 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:04:32.989 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:04:32.989 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:04:32.989 INFO:tasks.workunit.client.0.trial112.stderr:149392: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:04:32.989 INFO:tasks.workunit.client.0.trial112.stderr:149392: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:04:32.989 INFO:tasks.workunit.client.0.trial112.stderr:149392: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:04:32.989 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:04:32.989 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:04:32.989 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:04:32.989 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:04:32.989 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:04:32.989 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:04:32.989 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:04:32.989 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:04:32.989 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:04:32.989 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:04:32.989 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:04:32.989 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2026-02-13T02:04:32.989 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-13T02:04:32.989 INFO:tasks.workunit.client.0.trial112.stderr:149392: start osd.2 2026-02-13T02:04:32.989 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:04:34.448 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:04:34.448 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:04:34.449 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:04:34.449 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:04:34.449 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:04:34.449 INFO:tasks.workunit.client.0.trial112.stderr:149389: 2026-02-13T02:04:32.970+0000 7f4a9c1dca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:04:34.449 INFO:tasks.workunit.client.0.trial112.stderr:149389: 2026-02-13T02:04:32.970+0000 7f4a9c1dca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:04:34.449 INFO:tasks.workunit.client.0.trial112.stderr:149389: 2026-02-13T02:04:32.970+0000 7f4a9c1dca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:04:34.449 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:04:34.449 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:04:34.449 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:04:34.449 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:04:34.449 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:04:34.450 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:04:34.450 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:04:34.450 INFO:tasks.workunit.client.0.trial112.stderr:149389: 0 2026-02-13T02:04:34.450 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:04:34.450 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:04:34.450 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:04:34.450 INFO:tasks.workunit.client.0.trial112.stderr:149389: 2026-02-13T02:04:33.470+0000 7f4a9c1dca40 -1 Falling back to public interface 2026-02-13T02:04:34.450 INFO:tasks.workunit.client.0.trial112.stderr:149389: 2026-02-13T02:04:34.126+0000 7f4a9c1dca40 -1 osd.1 59 log_to_monitors true 2026-02-13T02:04:34.450 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:04:34.450 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:04:34.450 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:04:34.450 INFO:tasks.workunit.client.0.trial112.stderr:149389: 1 2026-02-13T02:04:34.451 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:04:34.451 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:04:34.451 INFO:tasks.workunit.client.0.trial112.stderr:149389: osd.1 up in weight 1 up_from 62 up_thru 62 down_at 59 last_clean_interval [50,59) [v2:127.0.0.1:6818/2349622718,v1:127.0.0.1:6819/2349622718] [v2:127.0.0.1:6820/2349622718,v1:127.0.0.1:6821/2349622718] exists,up e2e6d346-242c-41aa-a5a5-fbdd15bada25 2026-02-13T02:04:34.451 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:04:34.451 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:04:34.451 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:04:34.451 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:04:34.451 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:04:34.451 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:04:34.451 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:04:34.451 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:04:34.452 INFO:tasks.workunit.client.0.trial112.stderr:149389: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:04:34.484 INFO:tasks.workunit.client.0.trial112.stderr:149389: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: g49392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2026-02-13T02:04:34.485 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-13T02:04:34.485 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:04:34.485 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:04:34.485 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:04:34.485 INFO:tasks.workunit.client.0.trial112.stderr:149392: 2026-02-13T02:04:32.994+0000 7fa919dcba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:04:34.485 INFO:tasks.workunit.client.0.trial112.stderr:149392: 2026-02-13T02:04:32.994+0000 7fa919dcba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:04:34.485 INFO:tasks.workunit.client.0.trial112.stderr:149392: 2026-02-13T02:04:32.998+0000 7fa919dcba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:04:34.485 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-13T02:04:34.485 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:04:34.485 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T02:04:34.485 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:04:34.485 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:04:34.486 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:04:34.486 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:04:34.486 INFO:tasks.workunit.client.0.trial112.stderr:149392: 0 2026-02-13T02:04:34.486 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:04:34.486 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:04:34.486 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:04:34.486 INFO:tasks.workunit.client.0.trial112.stderr:149392: 2026-02-13T02:04:33.462+0000 7fa919dcba40 -1 Falling back to public interface 2026-02-13T02:04:34.486 INFO:tasks.workunit.client.0.trial112.stderr:149392: 2026-02-13T02:04:34.130+0000 7fa919dcba40 -1 osd.2 58 log_to_monitors true 2026-02-13T02:04:34.486 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:04:34.486 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:04:34.486 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:04:34.486 INFO:tasks.workunit.client.0.trial112.stderr:149392: 1 2026-02-13T02:04:34.487 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:04:34.487 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:04:34.487 INFO:tasks.workunit.client.0.trial112.stderr:149392: osd.2 up in weight 1 up_from 62 up_thru 47 down_at 59 last_clean_interval [15,58) [v2:127.0.0.1:6810/3469685006,v1:127.0.0.1:6811/3469685006] [v2:127.0.0.1:6812/3469685006,v1:127.0.0.1:6813/3469685006] exists,up 28a899a3-d0e3-4eb8-b652-37d882ac97d0 2026-02-13T02:04:34.487 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:04:34.487 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:04:34.487 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:04:34.487 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:04:34.487 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:04:34.487 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:04:34.487 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:04:34.487 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:04:34.487 INFO:tasks.workunit.client.0.trial112.stderr:149392: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:04:35.004 INFO:tasks.workunit.client.0.trial112.stderr:149392: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:04:35.004 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:04:35.004 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:04:35.004 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:04:35.004 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:04:35.004 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:04:35.004 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:04:35.004 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:04:35.004 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:04:35.004 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:04:35.004 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:04:35.004 INFO:tasks.workunit.client.0.trial112.stderr:149389: 1 2026-02-13T02:04:35.004 INFO:tasks.workunit.client.0.trial112.stderr:149389: 2 2026-02-13T02:04:35.004 INFO:tasks.workunit.client.0.trial112.stderr:149389: 3' 2026-02-13T02:04:35.004 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:04:35.004 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:04:35.004 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:04:35.004 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836499 2026-02-13T02:04:35.004 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836499 2026-02-13T02:04:35.004 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499' 2026-02-13T02:04:35.004 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:04:35.004 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:04:35.004 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=266287972354 2026-02-13T02:04:35.004 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 266287972354 2026-02-13T02:04:35.004 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-266287972354' 2026-02-13T02:04:35.004 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:04:35.004 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:04:35.004 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=266287972354 2026-02-13T02:04:35.004 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 266287972354 2026-02-13T02:04:35.004 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-266287972354 2-266287972354' 2026-02-13T02:04:35.004 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:04:35.004 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:04:35.004 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=240518168580 2026-02-13T02:04:35.004 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 240518168580 2026-02-13T02:04:35.005 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-266287972354 2-266287972354 3-240518168580' 2026-02-13T02:04:35.005 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:35.005 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836499 2026-02-13T02:04:35.050 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/et_timeout_delays: echo true 2026-02-13T02:04:35.050 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:04:35.050 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:04:35.050 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:04:35.051 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:04:35.051 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:04:35.051 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:04:35.051 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:04:35.051 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:04:35.051 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:04:35.051 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:04:35.051 INFO:tasks.workunit.client.0.trial112.stderr:149392: 1 2026-02-13T02:04:35.051 INFO:tasks.workunit.client.0.trial112.stderr:149392: 2 2026-02-13T02:04:35.051 INFO:tasks.workunit.client.0.trial112.stderr:149392: 3' 2026-02-13T02:04:35.051 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:04:35.052 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:04:35.052 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:04:35.052 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836500 2026-02-13T02:04:35.052 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836500 2026-02-13T02:04:35.052 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836500' 2026-02-13T02:04:35.052 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:04:35.052 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:04:35.052 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=266287972355 2026-02-13T02:04:35.052 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 266287972355 2026-02-13T02:04:35.052 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836500 1-266287972355' 2026-02-13T02:04:35.052 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:04:35.052 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:04:35.052 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=266287972355 2026-02-13T02:04:35.053 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 266287972355 2026-02-13T02:04:35.053 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836500 1-266287972355 2-266287972355' 2026-02-13T02:04:35.053 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:04:35.053 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:04:35.053 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=240518168581 2026-02-13T02:04:35.053 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 240518168581 2026-02-13T02:04:35.053 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836500 1-266287972355 2-266287972355 3-240518168581' 2026-02-13T02:04:35.053 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:35.053 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836500 2026-02-13T02:04:35.510 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:35.510 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:04:35.510 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836499 2026-02-13T02:04:35.510 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:04:35.510 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836499 2026-02-13T02:04:35.511 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2026-02-13T02:04:35.511 INFO:tasks.workunit.client.0.trial112.stderr:149389: waiting osd.0 seq 21474836499 2026-02-13T02:04:35.511 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:04:35.511 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836500 -lt 21474836499 2026-02-13T02:04:35.511 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:35.511 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-266287972354 2026-02-13T02:04:35.511 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:35.511 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:04:35.511 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-266287972354 2026-02-13T02:04:35.511 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:04:35.511 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=266287972354 2026-02-13T02:04:35.511 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 266287972354' 2026-02-13T02:04:35.512 INFO:tasks.workunit.client.0.trial112.stderr:149389: waiting osd.1 seq 266287972354 2026-02-13T02:04:35.512 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:04:35.512 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 266287972355 -lt 266287972354 2026-02-13T02:04:35.512 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:35.512 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-266287972354 2026-02-13T02:04:35.512 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:35.512 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:04:35.512 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-266287972354 2026-02-13T02:04:35.512 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:04:35.512 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=266287972354 2026-02-13T02:04:35.512 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 266287972354' 2026-02-13T02:04:35.512 INFO:tasks.workunit.client.0.trial112.stderr:149389: waiting osd.2 seq 266287972354 2026-02-13T02:04:35.513 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:04:35.513 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 266287972354 -lt 266287972354 2026-02-13T02:04:35.513 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:35.513 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-240518168580 2026-02-13T02:04:35.513 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:35.513 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:04:35.513 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-240518168580 2026-02-13T02:04:37.705 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats:ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:37.705 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:04:37.705 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836500 2026-02-13T02:04:37.706 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:04:37.706 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836500 2026-02-13T02:04:37.706 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2026-02-13T02:04:37.706 INFO:tasks.workunit.client.0.trial112.stderr:149392: waiting osd.0 seq 21474836500 2026-02-13T02:04:37.706 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:04:37.706 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836500 -lt 21474836500 2026-02-13T02:04:37.706 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:37.706 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-266287972355 2026-02-13T02:04:37.706 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:37.706 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:04:37.706 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-266287972355 2026-02-13T02:04:37.707 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:04:37.707 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=266287972355 2026-02-13T02:04:37.707 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 266287972355' 2026-02-13T02:04:37.707 INFO:tasks.workunit.client.0.trial112.stderr:149392: waiting osd.1 seq 266287972355 2026-02-13T02:04:37.707 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:04:37.707 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 266287972355 -lt 266287972355 2026-02-13T02:04:37.707 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:37.707 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-266287972355 2026-02-13T02:04:37.707 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:37.707 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:04:37.707 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-266287972355 2026-02-13T02:04:37.707 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:04:37.707 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=266287972355 2026-02-13T02:04:37.708 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 266287972355' 2026-02-13T02:04:37.708 INFO:tasks.workunit.client.0.trial112.stderr:149392: waiting osd.2 seq 266287972355 2026-02-13T02:04:37.708 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:04:37.708 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 266287972354 -lt 266287972355 2026-02-13T02:04:37.708 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:04:37.708 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:04:37.708 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:04:37.708 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 266287972354 -lt 266287972355 2026-02-13T02:04:37.708 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:04:38.550 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: cut -d - -f 2 2026-02-13T02:04:38.550 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=240518168580 2026-02-13T02:04:38.551 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 240518168580' 2026-02-13T02:04:38.551 INFO:tasks.workunit.client.0.trial112.stderr:149389: waiting osd.3 seq 240518168580 2026-02-13T02:04:38.551 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:04:38.551 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 240518168579 -lt 240518168580 2026-02-13T02:04:38.551 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:04:38.551 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:04:38.551 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:04:38.551 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 240518168579 -lt 240518168580 2026-02-13T02:04:38.551 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:04:38.551 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:04:38.551 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:04:38.551 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 240518168581 -lt 240518168580 2026-02-13T02:04:38.551 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:04:38.551 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:04:38.551 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:04:38.551 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:04:38.551 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:04:38.551 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:04:38.551 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:04:38.551 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:04:38.551 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:04:38.551 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:04:38.551 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:04:38.551 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:04:38.551 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:04:38.551 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:04:38.551 INFO:tasks.workunit.client.0.trial112.stderr:149389: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:04:38.551 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:04:38.551 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:04:38.551 INFO:tasks.workunit.client.0.trial112.stderr:149389: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:04:38.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-13T02:04:38.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-13T02:04:38.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 149388 2026-02-13T02:04:38.607 INFO:tasks.workunit.client.0.trial112.stderr: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:04:38.608 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:04:38.608 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 266287972355 -lt 266287972355 2026-02-13T02:04:38.608 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:38.608 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-240518168581 2026-02-13T02:04:38.608 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:38.608 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:04:38.608 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-240518168581 2026-02-13T02:04:38.608 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:04:38.608 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=240518168581 2026-02-13T02:04:38.608 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 240518168581' 2026-02-13T02:04:38.608 INFO:tasks.workunit.client.0.trial112.stderr:149392: waiting osd.3 seq 240518168581 2026-02-13T02:04:38.608 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:04:38.608 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 240518168581 -lt 240518168581 2026-02-13T02:04:38.608 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:04:38.608 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:04:38.608 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:04:38.608 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:04:38.608 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:04:38.608 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:04:38.608 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:04:38.608 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:04:38.608 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:04:38.608 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:04:38.608 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:04:38.608 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:04:38.608 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:04:38.608 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:04:38.608 INFO:tasks.workunit.client.0.trial112.stderr:149392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:04:38.608 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:04:38.608 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:04:38.608 INFO:tasks.workunit.client.0.trial112.stderr:149392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:04:38.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-13T02:04:38.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-13T02:04:38.609 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-13T02:04:38.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-13T02:04:38.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:257: corrupt_and_repair_two: return_code=0 2026-02-13T02:04:38.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:258: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-13T02:04:38.609 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: get_pg ecpool SOMETHING 2026-02-13T02:04:38.609 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-13T02:04:38.609 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-13T02:04:38.609 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-13T02:04:38.609 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T02:04:38.774 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: local pg=2.0 2026-02-13T02:04:38.774 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:264: corrupt_and_repair_two: repair 2.0 2026-02-13T02:04:38.774 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-13T02:04:38.774 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-13T02:04:38.774 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:04:38.774 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:04:38.774 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:04:38.774 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:04:38.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-13T02:04:22.198682+0000 2026-02-13T02:04:38.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-13T02:04:39.089 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-13T02:04:39.099 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-13T02:04:22.198682+0000 2026-02-13T02:04:39.099 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-13T02:04:39.099 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:04:22.198682+0000 2026-02-13T02:04:39.099 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:04:39.099 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:04:39.099 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:04:39.099 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:04:39.099 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:04:39.100 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:04:39.100 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:04:39.100 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:04:39.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:04:22.198682+0000 '>' 2026-02-13T02:04:22.198682+0000 2026-02-13T02:04:39.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:04:40.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:04:40.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:04:40.263 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:04:40.263 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:04:40.263 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:04:40.263 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:04:40.264 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:04:40.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:04:22.198682+0000 '>' 2026-02-13T02:04:22.198682+0000 2026-02-13T02:04:40.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:04:41.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:04:41.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:04:41.425 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:04:41.425 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:04:41.425 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:04:41.426 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:04:41.426 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:04:41.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:04:22.198682+0000 '>' 2026-02-13T02:04:22.198682+0000 2026-02-13T02:04:41.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:04:42.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:04:42.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:04:42.584 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:04:42.585 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:04:42.585 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:04:42.585 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:04:42.585 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:04:42.748 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:04:22.198682+0000 '>' 2026-02-13T02:04:22.198682+0000 2026-02-13T02:04:42.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:04:43.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:04:43.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:04:43.751 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:04:43.751 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:04:43.751 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:04:43.751 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:04:43.752 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:04:43.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:04:22.198682+0000 '>' 2026-02-13T02:04:22.198682+0000 2026-02-13T02:04:43.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:04:44.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:04:44.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:04:44.918 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:04:44.919 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:04:44.919 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:04:44.919 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:04:44.919 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:04:45.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:04:39.848962+0000 '>' 2026-02-13T02:04:22.198682+0000 2026-02-13T02:04:45.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:04:45.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: corrupt_and_repair_two: pids= 2026-02-13T02:04:45.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-13T02:04:45.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-13T02:04:45.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-13T02:04:45.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 152269"' 2026-02-13T02:04:45.083 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 152269' 2026-02-13T02:04:45.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2026-02-13T02:04:45.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-13T02:04:45.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-13T02:04:45.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 152270"' 2026-02-13T02:04:45.084 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 152270' 2026-02-13T02:04:45.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:271: corrupt_and_repair_two: wait_background pids 2026-02-13T02:04:45.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 152269 152270' 2026-02-13T02:04:45.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-13T02:04:45.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-13T02:04:45.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 152269 2026-02-13T02:04:45.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-13T02:04:45.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/152272: /' 2026-02-13T02:04:45.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2026-02-13T02:04:45.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/152274: /' 2026-02-13T02:04:45.561 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:04:45.561 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:04:45.561 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:04:45.562 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:04:45.562 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-13T02:04:45.562 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:04:45.562 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:04:45.562 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:04:45.562 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:04:45.562 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:04:45.562 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:04:45.562 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:04:45.562 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:04:45.562 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:04:45.563 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:04:45.563 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:04:45.563 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-13T02:04:45.563 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:04:45.563 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:04:45.563 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:04:45.563 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:04:45.563 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:04:45.563 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2026-02-13T02:04:45.563 INFO:tasks.workunit.client.0.trial112.stderr:152272: _ 2026-02-13T02:04:45.563 INFO:tasks.workunit.client.0.trial112.stderr:152272: hinfo_key 2026-02-13T02:04:45.563 INFO:tasks.workunit.client.0.trial112.stderr:152272: snapset 2026-02-13T02:04:45.564 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-13T02:04:45.564 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:04:45.564 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:04:45.564 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:04:45.564 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:04:45.564 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:04:45.564 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:04:45.564 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:04:45.565 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:04:45.565 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:04:45.565 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-13T02:04:45.565 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:04:45.565 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2026-02-13T02:04:45.566 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:04:45.566 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:04:45.566 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-13T02:04:45.566 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:04:45.566 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2026-02-13T02:04:45.566 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:04:45.566 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:04:45.566 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:04:45.566 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:04:45.566 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:04:45.566 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:04:45.566 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2026-02-13T02:04:45.567 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:04:45.567 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:04:45.567 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-13T02:04:45.567 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:04:45.567 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2026-02-13T02:04:45.567 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2026-02-13T02:04:45.567 INFO:tasks.workunit.client.0.trial112.stderr:152274: _ 2026-02-13T02:04:45.567 INFO:tasks.workunit.client.0.trial112.stderr:152274: hinfo_key 2026-02-13T02:04:45.567 INFO:tasks.workunit.client.0.trial112.stderr:152274: snapset 2026-02-13T02:04:45.567 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2026-02-13T02:04:45.567 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:04:45.567 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:04:45.568 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-13T02:04:45.568 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:04:45.568 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2026-02-13T02:04:45.568 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:04:45.568 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:04:45.568 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:04:45.568 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:04:45.568 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:04:45.568 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:04:45.568 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:04:45.568 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:04:45.568 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:04:45.569 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:04:45.569 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:04:45.569 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:04:45.569 INFO:tasks.workunit.client.0.trial112.stderr:152272: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:04:45.569 INFO:tasks.workunit.client.0.trial112.stderr:152272: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:04:45.569 INFO:tasks.workunit.client.0.trial112.stderr:152272: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:04:45.569 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:04:45.569 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:04:45.569 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:04:45.569 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:04:45.569 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:04:45.569 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:04:45.569 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:04:45.570 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:04:45.570 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:04:45.570 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:04:45.570 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:04:45.570 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:04:45.570 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:04:45.570 INFO:tasks.workunit.client.0.trial112.stderr:152272: start osd.1 2026-02-13T02:04:45.570 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recos.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:04:45.570 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:04:45.570 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-13T02:04:45.570 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-13T02:04:45.570 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:04:45.571 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:04:45.571 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:04:45.571 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:04:45.571 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:04:45.571 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:04:45.571 INFO:tasks.workunit.client.0.trial112.stderr:152274: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:04:45.571 INFO:tasks.workunit.client.0.trial112.stderr:152274: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:04:45.571 INFO:tasks.workunit.client.0.trial112.stderr:152274: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:04:45.571 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:04:45.571 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:04:45.571 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:04:45.571 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:04:45.572 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:04:45.572 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:04:45.572 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:04:45.572 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:04:45.572 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:04:45.572 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:04:45.572 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:04:45.572 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2026-02-13T02:04:45.573 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-13T02:04:45.573 INFO:tasks.workunit.client.0.trial112.stderr:152274: start osd.2 2026-02-13T02:04:47.051 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:04:47.051 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:04:47.051 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:04:47.051 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:04:47.051 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:04:47.051 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:04:47.051 INFO:tasks.workunit.client.0.trial112.stderr:152272: 2026-02-13T02:04:45.570+0000 7f12a1218a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:04:47.051 INFO:tasks.workunit.client.0.trial112.stderr:152272: 2026-02-13T02:04:45.574+0000 7f12a1218a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:04:47.051 INFO:tasks.workunit.client.0.trial112.stderr:152272: 2026-02-13T02:04:45.574+0000 7f12a1218a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:04:47.052 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:04:47.052 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:04:47.052 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:04:47.052 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:04:47.052 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:04:47.052 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:04:47.052 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:04:47.052 INFO:tasks.workunit.client.0.trial112.stderr:152272: 0 2026-02-13T02:04:47.052 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:04:47.052 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:04:47.052 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:04:47.053 INFO:tasks.workunit.client.0.trial112.stderr:152272: 2026-02-13T02:04:46.042+0000 7f12a1218a40 -1 Falling back to public interface 2026-02-13T02:04:47.053 INFO:tasks.workunit.client.0.trial112.stderr:152272: 2026-02-13T02:04:46.698+0000 7f12a1218a40 -1 osd.1 65 log_to_monitors true 2026-02-13T02:04:47.053 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:04:47.053 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:04:47.053 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:04:47.053 INFO:tasks.workunit.client.0.trial112.stderr:152272: 1 2026-02-13T02:04:47.053 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:04:47.053 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:04:47.053 INFO:tasks.workunit.client.0.trial112.stderr:152272: osd.1 up in weight 1 up_from 69 up_thru 69 down_at 66 last_clean_interval [62,65) [v2:127.0.0.1:6818/546305129,v1:127.0.0.1:6819/546305129] [v2:127.0.0.1:6820/546305129,v1:127.0.0.1:6821/546305129] exists,up e2e6d346-242c-41aa-a5a5-fbdd15bada25 2026-02-13T02:04:47.053 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:04:47.053 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:04:47.053 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:04:47.054 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:04:47.054 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:04:47.054 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:04:47.054 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:04:47.054 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:04:47.054 INFO:tasks.workunit.client.0.trial112.stderr:152272: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:04:47.054 INFO:tasks.workunit.client.0.trial112.stderr:152272: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1very_ops 2026-02-13T02:04:47.055 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2026-02-13T02:04:47.055 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-13T02:04:47.055 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:04:47.055 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:04:47.055 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:04:47.055 INFO:tasks.workunit.client.0.trial112.stderr:152274: 2026-02-13T02:04:45.574+0000 7f0491984a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:04:47.055 INFO:tasks.workunit.client.0.trial112.stderr:152274: 2026-02-13T02:04:45.574+0000 7f0491984a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:04:47.055 INFO:tasks.workunit.client.0.trial112.stderr:152274: 2026-02-13T02:04:45.574+0000 7f0491984a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:04:47.055 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-13T02:04:47.055 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:04:47.055 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T02:04:47.055 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:04:47.056 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:04:47.056 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:04:47.056 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:04:47.056 INFO:tasks.workunit.client.0.trial112.stderr:152274: 0 2026-02-13T02:04:47.056 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:04:47.056 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:04:47.056 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:04:47.056 INFO:tasks.workunit.client.0.trial112.stderr:152274: 2026-02-13T02:04:46.030+0000 7f0491984a40 -1 Falling back to public interface 2026-02-13T02:04:47.056 INFO:tasks.workunit.client.0.trial112.stderr:152274: 2026-02-13T02:04:46.682+0000 7f0491984a40 -1 osd.2 65 log_to_monitors true 2026-02-13T02:04:47.056 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:04:47.056 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:04:47.056 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:04:47.056 INFO:tasks.workunit.client.0.trial112.stderr:152274: 1 2026-02-13T02:04:47.057 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:04:47.057 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:04:47.057 INFO:tasks.workunit.client.0.trial112.stderr:152274: osd.2 up in weight 1 up_from 69 up_thru 47 down_at 66 last_clean_interval [62,65) [v2:127.0.0.1:6810/1663451615,v1:127.0.0.1:6811/1663451615] [v2:127.0.0.1:6812/1663451615,v1:127.0.0.1:6813/1663451615] exists,up 28a899a3-d0e3-4eb8-b652-37d882ac97d0 2026-02-13T02:04:47.057 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:04:47.057 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:04:47.057 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:04:47.057 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:04:47.057 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:04:47.057 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:04:47.057 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:04:47.058 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:04:47.058 INFO:tasks.workunit.client.0.trial112.stderr:152274: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:04:47.616 INFO:tasks.workunit.client.0.trial112.stderr:152274: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:04:47.616 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:04:47.616 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:04:47.616 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:04:47.616 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:04:47.617 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:04:47.617 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:04:47.617 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:04:47.617 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:04:47.617 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:04:47.617 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:04:47.617 INFO:tasks.workunit.client.0.trial112.stderr:152274: 1 2026-02-13T02:04:47.617 INFO:tasks.workunit.client.0.trial112.stderr:152274: 2 2026-02-13T02:04:47.617 INFO:tasks.workunit.client.0.trial112.stderr:152274: 3' 2026-02-13T02:04:47.617 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:04:47.617 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:04:47.617 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:04:47.618 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836503 2026-02-13T02:04:47.618 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836503 2026-02-13T02:04:47.618 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836503' 2026-02-13T02:04:47.618 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:04:47.618 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:04:47.618 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743426 2026-02-13T02:04:47.618 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743426 2026-02-13T02:04:47.618 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836503 1-296352743426' 2026-02-13T02:04:47.618 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:04:47.618 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:04:47.618 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743426 2026-02-13T02:04:47.618 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743426 2026-02-13T02:04:47.619 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836503 1-296352743426 2-296352743426' 2026-02-13T02:04:47.619 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:04:47.619 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:04:47.619 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=240518168585 2026-02-13T02:04:47.619 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 240518168585 2026-02-13T02:04:47.619 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836503 1-296352743426 2-296352743426 3-240518168585' 2026-02-13T02:04:47.619 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:47.619 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836503 2026-02-13T02:04:47.619 INFO:tasks.workunit.client.0.trial112.stderr:15227593: get_timeout_delays: echo true 2026-02-13T02:04:47.619 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:04:47.619 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:04:47.619 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:04:47.619 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:04:47.620 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:04:47.620 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:04:47.620 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:04:47.620 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:04:47.620 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:04:47.620 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:04:47.620 INFO:tasks.workunit.client.0.trial112.stderr:152272: 1 2026-02-13T02:04:47.620 INFO:tasks.workunit.client.0.trial112.stderr:152272: 2 2026-02-13T02:04:47.620 INFO:tasks.workunit.client.0.trial112.stderr:152272: 3' 2026-02-13T02:04:47.620 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:04:47.620 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:04:47.620 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:04:47.620 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836504 2026-02-13T02:04:47.621 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836504 2026-02-13T02:04:47.621 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504' 2026-02-13T02:04:47.621 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:04:47.621 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:04:47.621 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743427 2026-02-13T02:04:47.621 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743427 2026-02-13T02:04:47.621 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-296352743427' 2026-02-13T02:04:47.621 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:04:47.621 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:04:47.621 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743427 2026-02-13T02:04:47.621 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743427 2026-02-13T02:04:47.621 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-296352743427 2-296352743427' 2026-02-13T02:04:47.622 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:04:47.622 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:04:47.622 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=240518168584 2026-02-13T02:04:47.622 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 240518168584 2026-02-13T02:04:47.622 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-296352743427 2-296352743427 3-240518168584' 2026-02-13T02:04:47.622 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:47.622 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836504 2026-02-13T02:04:50.274 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:50.275 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:04:50.275 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836504 2026-02-13T02:04:50.275 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:04:50.275 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836504 2026-02-13T02:04:50.275 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2026-02-13T02:04:50.275 INFO:tasks.workunit.client.0.trial112.stderr:152272: waiting osd.0 seq 21474836504 2026-02-13T02:04:50.275 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:04:50.275 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836502 -lt 21474836504 2026-02-13T02:04:50.275 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:04:50.276 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:04:50.276 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:04:50.276 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836502 -lt 21474836504 2026-02-13T02:04:50.276 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:04:50.276 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:04:50.276 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:04:50.276 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836505 -lt 21474836504 2026-02-13T02:04:50.276 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:50.276 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-296352743427 2026-02-13T02:04:50.276 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:50.276 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:04:50.276 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-296352743427 2026-02-13T02:04:50.277 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:04:50.277 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=296352743427 2026-02-13T02:04:50.277 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 296352743427' 2026-02-13T02:04:50.277 INFO:tasks.workunit.client.0.trial112.stderr:152272: waiting osd.1 seq 296352743427 2026-02-13T02:04:50.277 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:04:50.277 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 296352743427 -lt 296352743427 2026-02-13T02:04:50.277 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:50.277 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-296352743427 2026-02-13T02:04:50.277 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:50.277 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:04:50.277 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-296352743427 2026-02-13T02:04:50.277 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:04:50.278 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=296352743427 2026-02-13T02:04:50.278 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 296352743427' 2026-02-13T02:04:50.282 INFO:tasks.workunit.client.0.trial112.stderr:14: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:50.282 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:04:50.282 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836503 2026-02-13T02:04:50.282 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:04:50.282 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836503 2026-02-13T02:04:50.282 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2026-02-13T02:04:50.283 INFO:tasks.workunit.client.0.trial112.stderr:152274: waiting osd.0 seq 21474836503 2026-02-13T02:04:50.283 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:04:50.283 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836502 -lt 21474836503 2026-02-13T02:04:50.283 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:04:50.283 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:04:50.283 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:04:50.283 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836502 -lt 21474836503 2026-02-13T02:04:50.283 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:04:50.283 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:04:50.283 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:04:50.283 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836505 -lt 21474836503 2026-02-13T02:04:50.284 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:50.284 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-296352743426 2026-02-13T02:04:50.284 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:50.284 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:04:50.284 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-296352743426 2026-02-13T02:04:50.284 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:04:50.284 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=296352743426 2026-02-13T02:04:50.284 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 296352743426' 2026-02-13T02:04:50.284 INFO:tasks.workunit.client.0.trial112.stderr:152274: waiting osd.1 seq 296352743426 2026-02-13T02:04:50.284 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:04:50.284 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 296352743427 -lt 296352743426 2026-02-13T02:04:50.284 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:50.285 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-296352743426 2026-02-13T02:04:50.285 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:50.285 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:04:50.285 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-296352743426 2026-02-13T02:04:50.285 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:04:50.285 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=296352743426 2026-02-13T02:04:51.141 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 2963527434252272: waiting osd.2 seq 296352743427 2026-02-13T02:04:51.141 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:04:51.141 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 296352743427 -lt 296352743427 2026-02-13T02:04:51.141 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:51.142 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-240518168584 2026-02-13T02:04:51.142 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:51.142 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:04:51.142 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-240518168584 2026-02-13T02:04:51.142 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:04:51.142 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=240518168584 2026-02-13T02:04:51.142 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 240518168584' 2026-02-13T02:04:51.142 INFO:tasks.workunit.client.0.trial112.stderr:152272: waiting osd.3 seq 240518168584 2026-02-13T02:04:51.142 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:04:51.142 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 240518168585 -lt 240518168584 2026-02-13T02:04:51.142 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:04:51.142 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:04:51.142 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:04:51.142 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:04:51.142 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:04:51.142 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:04:51.142 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:04:51.143 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:04:51.143 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:04:51.143 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:04:51.143 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:04:51.143 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=3 2026-02-13T02:04:51.143 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:04:51.143 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:04:51.143 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:04:51.143 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 3 = 5 2026-02-13T02:04:51.143 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 3 '!=' -1 2026-02-13T02:04:51.143 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-13T02:04:51.143 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=3 2026-02-13T02:04:51.143 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-13T02:04:51.186 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers6' 2026-02-13T02:04:51.186 INFO:tasks.workunit.client.0.trial112.stderr:152274: waiting osd.2 seq 296352743426 2026-02-13T02:04:51.186 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:04:51.186 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 296352743427 -lt 296352743426 2026-02-13T02:04:51.186 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:51.187 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-240518168585 2026-02-13T02:04:51.187 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:51.187 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:04:51.187 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-240518168585 2026-02-13T02:04:51.187 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:04:51.187 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=240518168585 2026-02-13T02:04:51.187 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 240518168585' 2026-02-13T02:04:51.187 INFO:tasks.workunit.client.0.trial112.stderr:152274: waiting osd.3 seq 240518168585 2026-02-13T02:04:51.187 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:04:51.187 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 240518168585 -lt 240518168585 2026-02-13T02:04:51.187 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:04:51.187 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:04:51.188 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:04:51.188 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:04:51.188 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:04:51.188 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:04:51.188 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:04:51.188 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:04:51.188 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:04:51.188 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:04:51.188 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:04:51.188 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=3 2026-02-13T02:04:51.188 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:04:51.188 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:04:51.189 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:04:51.189 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 3 = 5 2026-02-13T02:04:51.189 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 3 '!=' -1 2026-02-13T02:04:51.189 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-13T02:04:51.189 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=3 2026-02-13T02:04:51.189 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-13T02:04:52.004 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-hel.sh:1675: wait_for_clean: sleep 0.1 2026-02-13T02:04:52.004 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-13T02:04:52.004 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:04:52.004 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:04:52.004 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:04:52.004 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:04:52.004 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:04:52.005 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:04:52.005 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:04:52.005 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=3 2026-02-13T02:04:52.005 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:04:52.005 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:04:52.005 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:04:52.005 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 3 = 5 2026-02-13T02:04:52.005 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 3 '!=' 3 2026-02-13T02:04:52.005 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-13T02:04:52.005 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-13T02:04:52.005 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-13T02:04:52.006 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-13T02:04:52.006 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-13T02:04:52.006 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-13T02:04:52.006 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-13T02:04:52.006 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-13T02:04:52.006 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-13T02:04:52.006 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-13T02:04:52.006 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-13T02:04:52.006 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-13T02:04:52.006 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-13T02:04:52.006 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:04:52.006 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:04:52.006 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:04:52.056 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpepers.sh:1675: wait_for_clean: sleep 0.1 2026-02-13T02:04:52.056 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-13T02:04:52.056 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:04:52.056 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:04:52.056 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:04:52.056 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:04:52.056 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:04:52.056 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:04:52.057 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:04:52.057 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=3 2026-02-13T02:04:52.057 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:04:52.057 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:04:52.057 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:04:52.057 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 3 = 5 2026-02-13T02:04:52.057 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 3 '!=' 3 2026-02-13T02:04:52.057 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-13T02:04:52.057 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-13T02:04:52.057 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-13T02:04:52.057 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-13T02:04:52.057 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-13T02:04:52.058 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-13T02:04:52.058 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-13T02:04:52.058 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-13T02:04:52.058 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-13T02:04:52.058 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-13T02:04:52.058 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-13T02:04:52.058 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-13T02:04:52.058 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-13T02:04:52.058 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:04:52.058 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:04:52.058 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:04:52.974 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-hrs.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:04:52.974 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:04:52.974 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:04:52.974 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:04:52.974 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=3 2026-02-13T02:04:52.974 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:04:52.974 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:04:52.974 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:04:52.974 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 3 = 5 2026-02-13T02:04:52.975 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 3 '!=' 3 2026-02-13T02:04:52.975 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-13T02:04:52.975 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-13T02:04:52.975 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-13T02:04:52.975 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-13T02:04:52.975 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-13T02:04:52.975 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-13T02:04:52.975 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-13T02:04:52.975 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-13T02:04:52.975 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-13T02:04:52.975 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 2 >= 13 )) 2026-02-13T02:04:52.976 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-13T02:04:52.976 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.4 2026-02-13T02:04:52.976 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-13T02:04:52.976 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:04:52.976 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:04:52.976 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:04:52.976 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:04:52.976 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:04:52.976 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:04:53.013 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and conelpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:04:53.014 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:04:53.014 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:04:53.014 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:04:53.014 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=3 2026-02-13T02:04:53.014 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:04:53.014 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:04:53.014 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:04:53.014 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 3 = 5 2026-02-13T02:04:53.014 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 3 '!=' 3 2026-02-13T02:04:53.015 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-13T02:04:53.015 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-13T02:04:53.015 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-13T02:04:53.015 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-13T02:04:53.015 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-13T02:04:53.015 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-13T02:04:53.015 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-13T02:04:53.015 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-13T02:04:53.015 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-13T02:04:53.015 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 2 >= 13 )) 2026-02-13T02:04:53.015 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-13T02:04:53.015 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.4 2026-02-13T02:04:53.016 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-13T02:04:53.016 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:04:53.016 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:04:53.016 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:04:53.016 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:04:53.016 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:04:53.016 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:04:53.346 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") andtains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:04:53.346 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:04:53.346 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:04:53.347 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:04:53.347 INFO:tasks.workunit.client.0.trial112.stderr:152272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:04:53.347 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:04:53.347 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:04:53.347 INFO:tasks.workunit.client.0.trial112.stderr:152272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:04:53.347 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-13T02:04:53.348 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-13T02:04:53.348 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 152270 2026-02-13T02:04:53.363 INFO:tasks.workunit.client.0.trial112.stderr: contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:04:53.363 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:04:53.364 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:04:53.364 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:04:53.364 INFO:tasks.workunit.client.0.trial112.stderr:152274: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:04:53.364 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:04:53.364 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:04:53.364 INFO:tasks.workunit.client.0.trial112.stderr:152274: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:04:53.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-13T02:04:53.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-13T02:04:53.366 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-13T02:04:53.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-13T02:04:53.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:272: corrupt_and_repair_two: return_code=0 2026-02-13T02:04:53.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-13T02:04:53.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:275: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-13T02:04:53.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:276: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-13T02:04:53.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:323: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2026-02-13T02:04:53.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:245: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2026-02-13T02:04:53.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:246: corrupt_and_repair_two: local poolname=ecpool 2026-02-13T02:04:53.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:247: corrupt_and_repair_two: local first=3 2026-02-13T02:04:53.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_two: local second=1 2026-02-13T02:04:53.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_two: pids= 2026-02-13T02:04:53.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-13T02:04:53.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-13T02:04:53.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-13T02:04:53.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 154978"' 2026-02-13T02:04:53.388 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 154978' 2026-02-13T02:04:53.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2026-02-13T02:04:53.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-13T02:04:53.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-13T02:04:53.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 154980"' 2026-02-13T02:04:53.388 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 154980' 2026-02-13T02:04:53.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-13T02:04:53.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_two: wait_background pids 2026-02-13T02:04:53.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 154978 154980' 2026-02-13T02:04:53.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-13T02:04:53.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-13T02:04:53.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 154978 2026-02-13T02:04:53.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/154981: /' 2026-02-13T02:04:53.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2026-02-13T02:04:53.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/154983: /' 2026-02-13T02:04:53.981 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:04:53.981 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:04:53.981 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:04:53.981 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:04:53.981 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2026-02-13T02:04:53.982 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:04:53.982 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:04:53.982 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:04:53.982 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:04:53.982 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:04:53.982 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:04:53.982 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:04:53.982 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:04:53.982 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:04:53.982 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:04:53.982 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:04:53.982 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2026-02-13T02:04:53.982 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:04:53.982 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:04:53.982 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:04:53.982 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:04:53.982 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:04:53.982 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2026-02-13T02:04:53.983 INFO:tasks.workunit.client.0.trial112.stderr:154983: remove 1#2:eb822e21:::SOMETHING:head# 2026-02-13T02:04:53.983 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-13T02:04:53.983 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:04:53.983 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:04:53.983 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:04:53.983 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:04:53.983 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:04:53.983 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:04:53.983 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:04:53.985 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:04:53.985 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:04:53.985 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:04:53.985 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:04:53.985 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:04:53.985 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:04:53.985 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:04:53.985 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:04:53.985 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:04:53.985 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:04:53.986 INFO:tasks.workunit.client.0.trial112.stderr:154983: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:04:53.986 INFO:tasks.workunit.client.0.trial112.stderr:154983: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:04:53.986 INFO:tasks.workunit.client.0.trial112.stderr:154983: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:04:53.986 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:04:53.986 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:04:53.986 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:04:53.986 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:04:53.986 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:04:53.986 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:04:53.986 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:04:53.986 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:04:53.986 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:04:53.986 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:04:53.986 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:04:53.986 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:04:53.987 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:04:53.987 INFO:tasks.workunit.client.0.trial112.stderr:154983: start osd.1 2026-02-13T02:04:53.987 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:04:54.006 INFO:tasks.workunit.client.0.trial112.stderr:1154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:04:54.006 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:04:54.006 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-13T02:04:54.006 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:04:54.006 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2026-02-13T02:04:54.006 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:04:54.006 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:04:54.006 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-13T02:04:54.007 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:04:54.007 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-13T02:04:54.007 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:04:54.007 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:04:54.007 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:04:54.007 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:04:54.007 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:04:54.007 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:04:54.007 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2026-02-13T02:04:54.007 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:04:54.007 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:04:54.007 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-13T02:04:54.007 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:04:54.008 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:04:54.008 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2026-02-13T02:04:54.008 INFO:tasks.workunit.client.0.trial112.stderr:154981: remove 0#2:eb822e21:::SOMETHING:head# 2026-02-13T02:04:54.008 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-13T02:04:54.008 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:04:54.008 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:04:54.008 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-13T02:04:54.008 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:04:54.008 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:04:54.008 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:04:54.008 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:04:54.009 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:04:54.009 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:04:54.009 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-13T02:04:54.009 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-13T02:04:54.010 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:04:54.010 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:04:54.010 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:04:54.010 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:04:54.010 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:04:54.010 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:04:54.010 INFO:tasks.workunit.client.0.trial112.stderr:154981: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:04:54.010 INFO:tasks.workunit.client.0.trial112.stderr:154981: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:04:54.010 INFO:tasks.workunit.client.0.trial112.stderr:154981: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:04:54.010 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:04:54.010 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:04:54.010 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:04:54.011 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:04:54.011 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:04:54.011 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:04:54.011 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:04:54.011 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:04:54.011 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:04:54.011 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:04:54.011 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:04:54.011 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-13T02:04:54.011 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-13T02:04:54.011 INFO:tasks.workunit.client.0.trial112.stderr:154981: start osd.3 2026-02-13T02:04:54.011 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:04:55.468 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:04:55.469 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:04:55.469 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:04:55.469 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:04:55.469 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:04:55.469 INFO:tasks.workunit.client.0.trial112.stderr:154983: 2026-02-13T02:04:53.994+0000 7fcf8a9a5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:04:55.469 INFO:tasks.workunit.client.0.trial112.stderr:154983: 2026-02-13T02:04:53.994+0000 7fcf8a9a5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:04:55.469 INFO:tasks.workunit.client.0.trial112.stderr:154983: 2026-02-13T02:04:53.998+0000 7fcf8a9a5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:04:55.469 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:04:55.470 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:04:55.470 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:04:55.470 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:04:55.470 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:04:55.470 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:04:55.470 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:04:55.470 INFO:tasks.workunit.client.0.trial112.stderr:154983: 0 2026-02-13T02:04:55.470 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:04:55.470 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:04:55.470 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:04:55.470 INFO:tasks.workunit.client.0.trial112.stderr:154983: 2026-02-13T02:04:54.466+0000 7fcf8a9a5a40 -1 Falling back to public interface 2026-02-13T02:04:55.470 INFO:tasks.workunit.client.0.trial112.stderr:154983: 2026-02-13T02:04:55.106+0000 7fcf8a9a5a40 -1 osd.1 70 log_to_monitors true 2026-02-13T02:04:55.471 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:04:55.471 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:04:55.471 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:04:55.471 INFO:tasks.workunit.client.0.trial112.stderr:154983: 1 2026-02-13T02:04:55.471 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:04:55.471 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:04:55.471 INFO:tasks.workunit.client.0.trial112.stderr:154983: osd.1 up in weight 1 up_from 74 up_thru 74 down_at 71 last_clean_interval [69,70) [v2:127.0.0.1:6818/3806080108,v1:127.0.0.1:6819/3806080108] [v2:127.0.0.1:6820/3806080108,v1:127.0.0.1:6821/3806080108] exists,up e2e6d346-242c-41aa-a5a5-fbdd15bada25 2026-02-13T02:04:55.471 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:04:55.471 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:04:55.471 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:04:55.471 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:04:55.471 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:04:55.472 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:04:55.472 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:04:55.472 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:04:55.472 INFO:tasks.workunit.client.0.trial112.stderr:154983: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:04:55.499 INFO:tasks.workunit.client.0.trial112.stderr:154983: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: g54981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-13T02:04:55.500 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-13T02:04:55.500 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:04:55.500 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:04:55.500 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:04:55.500 INFO:tasks.workunit.client.0.trial112.stderr:154981: 2026-02-13T02:04:54.014+0000 7fed474efa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:04:55.500 INFO:tasks.workunit.client.0.trial112.stderr:154981: 2026-02-13T02:04:54.018+0000 7fed474efa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:04:55.500 INFO:tasks.workunit.client.0.trial112.stderr:154981: 2026-02-13T02:04:54.018+0000 7fed474efa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:04:55.500 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-13T02:04:55.500 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:04:55.500 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-13T02:04:55.500 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:04:55.501 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:04:55.501 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:04:55.501 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:04:55.501 INFO:tasks.workunit.client.0.trial112.stderr:154981: 0 2026-02-13T02:04:55.501 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:04:55.501 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:04:55.501 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:04:55.501 INFO:tasks.workunit.client.0.trial112.stderr:154981: 2026-02-13T02:04:54.486+0000 7fed474efa40 -1 Falling back to public interface 2026-02-13T02:04:55.501 INFO:tasks.workunit.client.0.trial112.stderr:154981: 2026-02-13T02:04:55.170+0000 7fed474efa40 -1 osd.3 70 log_to_monitors true 2026-02-13T02:04:55.501 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:04:55.501 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:04:55.501 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:04:55.502 INFO:tasks.workunit.client.0.trial112.stderr:154981: 1 2026-02-13T02:04:55.502 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:04:55.502 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:04:55.502 INFO:tasks.workunit.client.0.trial112.stderr:154981: osd.3 up in weight 1 up_from 75 up_thru 75 down_at 71 last_clean_interval [56,70) [v2:127.0.0.1:6826/611595779,v1:127.0.0.1:6827/611595779] [v2:127.0.0.1:6828/611595779,v1:127.0.0.1:6829/611595779] exists,up 4ffbcaae-5aa3-42b7-b511-2f4ca69d4da7 2026-02-13T02:04:55.502 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:04:55.502 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:04:55.502 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:04:55.502 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:04:55.502 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:04:55.502 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:04:55.502 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:04:55.502 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:04:55.502 INFO:tasks.workunit.client.0.trial112.stderr:154981: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:04:56.050 INFO:tasks.workunit.client.0.trial112.stderr:154981: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:04:56.050 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:04:56.050 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:04:56.050 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:04:56.050 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:04:56.050 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:04:56.050 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:04:56.051 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:04:56.051 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:04:56.051 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:04:56.051 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:04:56.051 INFO:tasks.workunit.client.0.trial112.stderr:154981: 1 2026-02-13T02:04:56.051 INFO:tasks.workunit.client.0.trial112.stderr:154981: 2 2026-02-13T02:04:56.051 INFO:tasks.workunit.client.0.trial112.stderr:154981: 3' 2026-02-13T02:04:56.051 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:04:56.051 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:04:56.051 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:04:56.051 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836508 2026-02-13T02:04:56.052 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836508 2026-02-13T02:04:56.052 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836508' 2026-02-13T02:04:56.052 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:04:56.052 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:04:56.052 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=317827579907 2026-02-13T02:04:56.052 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 317827579907 2026-02-13T02:04:56.052 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836508 1-317827579907' 2026-02-13T02:04:56.052 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:04:56.052 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:04:56.052 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743430 2026-02-13T02:04:56.052 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743430 2026-02-13T02:04:56.052 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836508 1-317827579907 2-296352743430' 2026-02-13T02:04:56.053 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:04:56.053 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:04:56.053 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=322122547202 2026-02-13T02:04:56.053 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 322122547202 2026-02-13T02:04:56.053 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836508 1-317827579907 2-296352743430 3-322122547202' 2026-02-13T02:04:56.053 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:56.053 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836508 2026-02-13T02:04:56.053 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubunet_timeout_delays: echo true 2026-02-13T02:04:56.054 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:04:56.054 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:04:56.054 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:04:56.054 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:04:56.054 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:04:56.054 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:04:56.054 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:04:56.054 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:04:56.054 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:04:56.054 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:04:56.054 INFO:tasks.workunit.client.0.trial112.stderr:154983: 1 2026-02-13T02:04:56.055 INFO:tasks.workunit.client.0.trial112.stderr:154983: 2 2026-02-13T02:04:56.055 INFO:tasks.workunit.client.0.trial112.stderr:154983: 3' 2026-02-13T02:04:56.055 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:04:56.055 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:04:56.055 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:04:56.055 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836507 2026-02-13T02:04:56.055 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836507 2026-02-13T02:04:56.055 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836507' 2026-02-13T02:04:56.055 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:04:56.055 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:04:56.055 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=317827579906 2026-02-13T02:04:56.055 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 317827579906 2026-02-13T02:04:56.056 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836507 1-317827579906' 2026-02-13T02:04:56.056 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:04:56.056 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:04:56.056 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743429 2026-02-13T02:04:56.056 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743429 2026-02-13T02:04:56.056 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836507 1-317827579906 2-296352743429' 2026-02-13T02:04:56.056 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:04:56.056 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:04:56.056 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=322122547203 2026-02-13T02:04:56.056 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 322122547203 2026-02-13T02:04:56.056 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836507 1-317827579906 2-296352743429 3-322122547203' 2026-02-13T02:04:56.056 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:56.056 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836507 2026-02-13T02:04:57.710 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/tu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:57.711 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:04:57.711 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836508 2026-02-13T02:04:57.711 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:04:57.711 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836508 2026-02-13T02:04:57.711 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2026-02-13T02:04:57.711 INFO:tasks.workunit.client.0.trial112.stderr:154981: waiting osd.0 seq 21474836508 2026-02-13T02:04:57.711 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:04:57.711 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836506 -lt 21474836508 2026-02-13T02:04:57.711 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:04:57.711 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:04:57.711 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:04:57.712 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836508 -lt 21474836508 2026-02-13T02:04:57.712 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:57.712 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-317827579907 2026-02-13T02:04:57.712 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:57.712 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:04:57.712 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-317827579907 2026-02-13T02:04:57.712 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:04:57.712 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=317827579907 2026-02-13T02:04:57.712 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 317827579907' 2026-02-13T02:04:57.712 INFO:tasks.workunit.client.0.trial112.stderr:154981: waiting osd.1 seq 317827579907 2026-02-13T02:04:57.712 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:04:57.712 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 317827579907 -lt 317827579907 2026-02-13T02:04:57.712 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:57.713 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-296352743430 2026-02-13T02:04:57.713 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:57.713 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:04:57.713 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-296352743430 2026-02-13T02:04:57.713 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:04:57.713 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=296352743430 2026-02-13T02:04:57.713 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 296352743430' 2026-02-13T02:04:57.713 INFO:tasks.workunit.client.0.trial112.stderr:154981: waiting osd.2 seq 296352743430 2026-02-13T02:04:57.713 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:04:57.713 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 296352743431 -lt 296352743430 2026-02-13T02:04:57.713 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:57.719 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:226ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:57.719 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:04:57.720 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836507 2026-02-13T02:04:57.720 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:04:57.720 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836507 2026-02-13T02:04:57.720 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2026-02-13T02:04:57.720 INFO:tasks.workunit.client.0.trial112.stderr:154983: waiting osd.0 seq 21474836507 2026-02-13T02:04:57.720 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:04:57.720 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836506 -lt 21474836507 2026-02-13T02:04:57.720 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:04:57.720 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:04:57.720 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:04:57.720 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836508 -lt 21474836507 2026-02-13T02:04:57.721 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:57.721 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-317827579906 2026-02-13T02:04:57.721 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:57.721 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:04:57.721 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-317827579906 2026-02-13T02:04:57.721 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:04:57.721 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=317827579906 2026-02-13T02:04:57.721 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 317827579906' 2026-02-13T02:04:57.721 INFO:tasks.workunit.client.0.trial112.stderr:154983: waiting osd.1 seq 317827579906 2026-02-13T02:04:57.721 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:04:57.721 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 317827579907 -lt 317827579906 2026-02-13T02:04:57.721 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:57.721 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-296352743429 2026-02-13T02:04:57.722 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:57.722 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:04:57.722 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-296352743429 2026-02-13T02:04:57.722 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:04:57.722 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=296352743429 2026-02-13T02:04:57.722 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 296352743429' 2026-02-13T02:04:57.722 INFO:tasks.workunit.client.0.trial112.stderr:154983: waiting osd.2 seq 296352743429 2026-02-13T02:04:57.722 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:04:57.722 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 296352743431 -lt 296352743429 2026-02-13T02:04:57.722 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:04:58.417 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh7: flush_pg_stats: echo 3-322122547202 2026-02-13T02:04:58.417 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:58.418 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:04:58.418 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-322122547202 2026-02-13T02:04:58.418 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:04:58.418 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=322122547202 2026-02-13T02:04:58.418 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 322122547202' 2026-02-13T02:04:58.418 INFO:tasks.workunit.client.0.trial112.stderr:154981: waiting osd.3 seq 322122547202 2026-02-13T02:04:58.418 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:04:58.418 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 322122547203 -lt 322122547202 2026-02-13T02:04:58.418 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:04:58.418 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:04:58.418 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:04:58.419 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:04:58.419 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:04:58.419 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:04:58.419 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:04:58.419 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:04:58.419 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:04:58.419 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:04:58.419 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:04:58.419 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:04:58.419 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:04:58.419 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:04:58.419 INFO:tasks.workunit.client.0.trial112.stderr:154981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:04:58.420 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:04:58.420 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:04:58.420 INFO:tasks.workunit.client.0.trial112.stderr:154981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:04:58.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-13T02:04:58.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-13T02:04:58.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 154980 2026-02-13T02:04:58.450 INFO:tasks.workunit.client.0.trial112.stderr::2267: flush_pg_stats: echo 3-322122547203 2026-02-13T02:04:58.450 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:04:58.450 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:04:58.450 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-322122547203 2026-02-13T02:04:58.450 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:04:58.450 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=322122547203 2026-02-13T02:04:58.450 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 322122547203' 2026-02-13T02:04:58.450 INFO:tasks.workunit.client.0.trial112.stderr:154983: waiting osd.3 seq 322122547203 2026-02-13T02:04:58.450 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:04:58.451 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 322122547203 -lt 322122547203 2026-02-13T02:04:58.451 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:04:58.451 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:04:58.451 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:04:58.451 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:04:58.451 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:04:58.451 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:04:58.451 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:04:58.451 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:04:58.451 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:04:58.451 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:04:58.451 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:04:58.451 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:04:58.452 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:04:58.452 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:04:58.452 INFO:tasks.workunit.client.0.trial112.stderr:154983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:04:58.452 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:04:58.452 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:04:58.452 INFO:tasks.workunit.client.0.trial112.stderr:154983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:04:58.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-13T02:04:58.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-13T02:04:58.454 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-13T02:04:58.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-13T02:04:58.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:257: corrupt_and_repair_two: return_code=0 2026-02-13T02:04:58.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:258: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-13T02:04:58.454 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: get_pg ecpool SOMETHING 2026-02-13T02:04:58.454 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-13T02:04:58.454 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-13T02:04:58.454 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-13T02:04:58.454 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T02:04:58.620 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: local pg=2.0 2026-02-13T02:04:58.620 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:264: corrupt_and_repair_two: repair 2.0 2026-02-13T02:04:58.620 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-13T02:04:58.621 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-13T02:04:58.621 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:04:58.621 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:04:58.621 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:04:58.621 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:04:58.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-13T02:04:39.848962+0000 2026-02-13T02:04:58.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-13T02:04:58.929 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-13T02:04:58.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-13T02:04:39.848962+0000 2026-02-13T02:04:58.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-13T02:04:58.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:04:39.848962+0000 2026-02-13T02:04:58.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:04:58.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:04:58.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:04:58.940 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:04:58.940 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:04:58.940 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:04:58.940 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:04:58.940 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:04:59.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:04:39.848962+0000 '>' 2026-02-13T02:04:39.848962+0000 2026-02-13T02:04:59.104 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:05:00.105 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:05:00.105 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:05:00.106 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:05:00.106 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:05:00.106 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:05:00.106 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:05:00.106 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:05:00.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:04:39.848962+0000 '>' 2026-02-13T02:04:39.848962+0000 2026-02-13T02:05:00.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:05:01.277 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:05:01.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:05:01.278 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:05:01.278 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:05:01.279 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:05:01.279 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:05:01.279 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:05:01.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:04:59.284294+0000 '>' 2026-02-13T02:04:39.848962+0000 2026-02-13T02:05:01.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:05:01.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: corrupt_and_repair_two: pids= 2026-02-13T02:05:01.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-13T02:05:01.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-13T02:05:01.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-13T02:05:01.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 157740"' 2026-02-13T02:05:01.446 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 157740' 2026-02-13T02:05:01.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-13T02:05:01.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-13T02:05:01.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-13T02:05:01.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 157742"' 2026-02-13T02:05:01.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-13T02:05:01.446 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 157742' 2026-02-13T02:05:01.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:271: corrupt_and_repair_two: wait_background pids 2026-02-13T02:05:01.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 157740 157742' 2026-02-13T02:05:01.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-13T02:05:01.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-13T02:05:01.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 157740 2026-02-13T02:05:01.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/157743: /' 2026-02-13T02:05:01.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-13T02:05:01.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/157745: /' 2026-02-13T02:05:01.973 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:05:01.973 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:05:01.973 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:05:01.974 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:05:01.974 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-13T02:05:01.974 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:05:01.974 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:05:01.974 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:05:01.974 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:05:01.974 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:05:01.974 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:05:01.974 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:05:01.974 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:05:01.974 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:05:01.974 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:05:01.975 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:05:01.975 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-13T02:05:01.975 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:05:01.975 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:05:01.975 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:05:01.975 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:05:01.975 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:05:01.975 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2026-02-13T02:05:01.975 INFO:tasks.workunit.client.0.trial112.stderr:157745: _ 2026-02-13T02:05:01.975 INFO:tasks.workunit.client.0.trial112.stderr:157745: hinfo_key 2026-02-13T02:05:01.975 INFO:tasks.workunit.client.0.trial112.stderr:157745: snapset 2026-02-13T02:05:01.976 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-13T02:05:01.976 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:05:01.976 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:05:01.976 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:05:01.976 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:05:01.976 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:05:01.976 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:05:01.976 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:05:01.977 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:05:01.977 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:05:01.977 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:05:01.977 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:05:01.977 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:05:01.977 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:05:01.977 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:05:01.977 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:05:01.977 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:05:01.978 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:05:01.978 INFO:tasks.workunit.client.0.trial112.stderr:157745: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:05:01.978 INFO:tasks.workunit.client.0.trial112.stderr:157745: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:05:01.978 INFO:tasks.workunit.client.0.trial112.stderr:157745: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:05:01.978 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:05:01.978 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:05:01.978 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:05:01.978 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:05:01.978 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:05:01.978 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:05:01.978 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:05:01.978 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:05:01.979 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:05:01.979 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:05:01.979 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:05:01.979 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:05:01.979 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:05:01.979 INFO:tasks.workunit.client.0.trial112.stderr:157745: start osd.1 2026-02-13T02:05:01.997 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reco157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:05:01.997 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:05:01.997 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-13T02:05:01.997 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:05:01.997 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-13T02:05:01.998 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:05:01.998 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:05:01.998 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-13T02:05:01.998 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:05:01.998 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-13T02:05:01.998 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:05:01.998 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:05:01.998 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:05:01.998 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:05:01.998 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:05:01.998 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:05:01.998 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-13T02:05:01.999 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:05:01.999 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:05:01.999 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-13T02:05:01.999 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:05:01.999 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:05:01.999 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2026-02-13T02:05:01.999 INFO:tasks.workunit.client.0.trial112.stderr:157743: Error getting attr on : 2.0s0_head,0#-4:00000000:::deep_scrub_2.0s0:head#, (61) No data available 2026-02-13T02:05:01.999 INFO:tasks.workunit.client.0.trial112.stderr:157743: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2026-02-13T02:05:01.999 INFO:tasks.workunit.client.0.trial112.stderr:157743: _ 2026-02-13T02:05:01.999 INFO:tasks.workunit.client.0.trial112.stderr:157743: hinfo_key 2026-02-13T02:05:01.999 INFO:tasks.workunit.client.0.trial112.stderr:157743: snapset 2026-02-13T02:05:01.999 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-13T02:05:02.000 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:05:02.000 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:05:02.000 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-13T02:05:02.000 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:05:02.000 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:05:02.000 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:05:02.000 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:05:02.001 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:05:02.001 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:05:02.001 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-13T02:05:02.001 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-13T02:05:02.001 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:05:02.001 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:05:02.001 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:05:02.001 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:05:02.001 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:05:02.001 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:05:02.001 INFO:tasks.workunit.client.0.trial112.stderr:157743: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:05:02.001 INFO:tasks.workunit.client.0.trial112.stderr:157743: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:05:02.002 INFO:tasks.workunit.client.0.trial112.stderr:157743: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:05:02.002 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:05:02.002 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:05:02.002 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:05:02.002 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:05:02.002 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:05:02.002 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:05:02.002 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:05:02.002 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:05:02.002 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:05:02.002 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:05:02.002 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:05:02.002 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-13T02:05:02.003 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-13T02:05:02.003 INFO:tasks.workunit.client.0.trial112.stderr:157743: start osd.3 2026-02-13T02:05:03.472 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.very_ops 2026-02-13T02:05:03.472 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:05:03.472 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:05:03.472 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:05:03.472 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:05:03.472 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:05:03.473 INFO:tasks.workunit.client.0.trial112.stderr:157745: 2026-02-13T02:05:01.986+0000 7fa524230a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:03.473 INFO:tasks.workunit.client.0.trial112.stderr:157745: 2026-02-13T02:05:01.986+0000 7fa524230a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:03.473 INFO:tasks.workunit.client.0.trial112.stderr:157745: 2026-02-13T02:05:01.990+0000 7fa524230a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:03.473 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:05:03.473 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:05:03.473 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:05:03.473 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:05:03.473 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:05:03.473 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:05:03.473 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:05:03.473 INFO:tasks.workunit.client.0.trial112.stderr:157745: 0 2026-02-13T02:05:03.473 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:05:03.474 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:05:03.474 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:05:03.474 INFO:tasks.workunit.client.0.trial112.stderr:157745: 2026-02-13T02:05:02.446+0000 7fa524230a40 -1 Falling back to public interface 2026-02-13T02:05:03.474 INFO:tasks.workunit.client.0.trial112.stderr:157745: 2026-02-13T02:05:03.109+0000 7fa524230a40 -1 osd.1 76 log_to_monitors true 2026-02-13T02:05:03.474 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:05:03.474 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:05:03.474 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:05:03.474 INFO:tasks.workunit.client.0.trial112.stderr:157745: 1 2026-02-13T02:05:03.474 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:05:03.474 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:05:03.474 INFO:tasks.workunit.client.0.trial112.stderr:157745: osd.1 up in weight 1 up_from 80 up_thru 80 down_at 77 last_clean_interval [74,76) [v2:127.0.0.1:6818/2871980403,v1:127.0.0.1:6819/2871980403] [v2:127.0.0.1:6820/2871980403,v1:127.0.0.1:6821/2871980403] exists,up e2e6d346-242c-41aa-a5a5-fbdd15bada25 2026-02-13T02:05:03.474 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:05:03.475 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:05:03.475 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:05:03.475 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:05:03.475 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:05:03.475 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:05:03.475 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:05:03.475 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:05:03.475 INFO:tasks.workunit.client.0.trial112.stderr:157745: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:05:03.495 INFO:tasks.workunit.client.0.trial112.stderr:157745: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:05:03.495 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-13T02:05:03.495 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-13T02:05:03.495 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:05:03.495 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:05:03.495 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:05:03.495 INFO:tasks.workunit.client.0.trial112.stderr:157743: 2026-02-13T02:05:02.006+0000 7f06abfd2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:03.495 INFO:tasks.workunit.client.0.trial112.stderr:157743: 2026-02-13T02:05:02.006+0000 7f06abfd2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:03.496 INFO:tasks.workunit.client.0.trial112.stderr:157743: 2026-02-13T02:05:02.006+0000 7f06abfd2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:03.496 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-13T02:05:03.496 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:05:03.496 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-13T02:05:03.496 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:05:03.496 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:05:03.496 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:05:03.496 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:05:03.496 INFO:tasks.workunit.client.0.trial112.stderr:157743: 0 2026-02-13T02:05:03.496 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:05:03.496 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:05:03.496 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:05:03.497 INFO:tasks.workunit.client.0.trial112.stderr:157743: 2026-02-13T02:05:02.466+0000 7f06abfd2a40 -1 Falling back to public interface 2026-02-13T02:05:03.497 INFO:tasks.workunit.client.0.trial112.stderr:157743: 2026-02-13T02:05:03.109+0000 7f06abfd2a40 -1 osd.3 76 log_to_monitors true 2026-02-13T02:05:03.497 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:05:03.497 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:05:03.497 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:05:03.497 INFO:tasks.workunit.client.0.trial112.stderr:157743: 1 2026-02-13T02:05:03.497 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:05:03.497 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:05:03.497 INFO:tasks.workunit.client.0.trial112.stderr:157743: osd.3 up in weight 1 up_from 80 up_thru 80 down_at 77 last_clean_interval [75,76) [v2:127.0.0.1:6826/2260098257,v1:127.0.0.1:6827/2260098257] [v2:127.0.0.1:6828/2260098257,v1:127.0.0.1:6829/2260098257] exists,up 4ffbcaae-5aa3-42b7-b511-2f4ca69d4da7 2026-02-13T02:05:03.497 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:05:03.497 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:05:03.498 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:05:03.498 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:05:03.498 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:05:03.498 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:05:03.498 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:05:04.022 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delsh:1593: get_timeout_delays: echo true 2026-02-13T02:05:04.022 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:05:04.022 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:05:04.022 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:05:04.022 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:05:04.022 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:05:04.022 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:05:04.023 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:05:04.023 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:05:04.023 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:05:04.023 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:05:04.023 INFO:tasks.workunit.client.0.trial112.stderr:157745: 1 2026-02-13T02:05:04.023 INFO:tasks.workunit.client.0.trial112.stderr:157745: 2 2026-02-13T02:05:04.023 INFO:tasks.workunit.client.0.trial112.stderr:157745: 3' 2026-02-13T02:05:04.023 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:05:04.023 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:05:04.023 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:05:04.023 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836511 2026-02-13T02:05:04.023 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836511 2026-02-13T02:05:04.023 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511' 2026-02-13T02:05:04.023 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:05:04.023 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:05:04.023 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=343597383682 2026-02-13T02:05:04.024 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 343597383682 2026-02-13T02:05:04.024 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511 1-343597383682' 2026-02-13T02:05:04.024 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:05:04.024 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:05:04.024 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743433 2026-02-13T02:05:04.024 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743433 2026-02-13T02:05:04.024 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511 1-343597383682 2-296352743433' 2026-02-13T02:05:04.024 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:05:04.024 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:05:04.024 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=343597383682 2026-02-13T02:05:04.024 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 343597383682 2026-02-13T02:05:04.024 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511 1-343597383682 2-296352743433 3-343597383682' 2026-02-13T02:05:04.024 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:05:04.024 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836511 2026-02-13T02:05:04.061 INFO:tasks.workunit.client.0.trial112.stderr:15774ays 90 .1 2026-02-13T02:05:04.061 INFO:tasks.workunit.client.0.trial112.stderr:157743: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:05:04.061 INFO:tasks.workunit.client.0.trial112.stderr:157743: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:05:04.061 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:05:04.061 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:05:04.061 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:05:04.062 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:05:04.062 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:05:04.062 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:05:04.062 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:05:04.062 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:05:04.062 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:05:04.062 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:05:04.062 INFO:tasks.workunit.client.0.trial112.stderr:157743: 1 2026-02-13T02:05:04.062 INFO:tasks.workunit.client.0.trial112.stderr:157743: 2 2026-02-13T02:05:04.062 INFO:tasks.workunit.client.0.trial112.stderr:157743: 3' 2026-02-13T02:05:04.062 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:05:04.062 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:05:04.062 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:05:04.062 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836512 2026-02-13T02:05:04.062 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836512 2026-02-13T02:05:04.062 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512' 2026-02-13T02:05:04.063 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:05:04.063 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:05:04.063 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=343597383683 2026-02-13T02:05:04.063 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 343597383683 2026-02-13T02:05:04.063 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512 1-343597383683' 2026-02-13T02:05:04.063 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:05:04.063 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:05:04.063 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743434 2026-02-13T02:05:04.063 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743434 2026-02-13T02:05:04.063 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512 1-343597383683 2-296352743434' 2026-02-13T02:05:04.063 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:05:04.063 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:05:04.063 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=343597383683 2026-02-13T02:05:04.063 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 343597383683 2026-02-13T02:05:04.063 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512 1-343597383683 2-296352743434 3-343597383683' 2026-02-13T02:05:05.693 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest5: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:05:05.693 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:05:05.693 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836511 2026-02-13T02:05:05.693 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:05:05.693 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836511 2026-02-13T02:05:05.694 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2026-02-13T02:05:05.694 INFO:tasks.workunit.client.0.trial112.stderr:157745: waiting osd.0 seq 21474836511 2026-02-13T02:05:05.694 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:05:05.694 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836509 -lt 21474836511 2026-02-13T02:05:05.694 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:05:05.694 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:05:05.694 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:05:05.694 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836512 -lt 21474836511 2026-02-13T02:05:05.694 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:05:05.694 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-343597383682 2026-02-13T02:05:05.694 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:05:05.694 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:05:05.695 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-343597383682 2026-02-13T02:05:05.695 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:05:05.695 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=343597383682 2026-02-13T02:05:05.695 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 343597383682' 2026-02-13T02:05:05.695 INFO:tasks.workunit.client.0.trial112.stderr:157745: waiting osd.1 seq 343597383682 2026-02-13T02:05:05.695 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:05:05.695 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 343597383683 -lt 343597383682 2026-02-13T02:05:05.695 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:05:05.695 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-296352743433 2026-02-13T02:05:05.695 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:05:05.695 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:05:05.695 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-296352743433 2026-02-13T02:05:05.695 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:05:05.696 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=296352743433 2026-02-13T02:05:05.696 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 296352743433' 2026-02-13T02:05:05.696 INFO:tasks.workunit.client.0.trial112.stderr:157745: waiting osd.2 seq 296352743433 2026-02-13T02:05:05.696 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:05:05.696 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 296352743434 -lt 296352743433 2026-02-13T02:05:05.696 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:05:05.765 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:05:05.765 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836512 2026-02-13T02:05:05.765 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:05:05.765 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:05:05.766 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836512 2026-02-13T02:05:05.766 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:05:05.766 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836512 2026-02-13T02:05:05.766 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2026-02-13T02:05:05.766 INFO:tasks.workunit.client.0.trial112.stderr:157743: waiting osd.0 seq 21474836512 2026-02-13T02:05:05.766 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:05:05.766 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836509 -lt 21474836512 2026-02-13T02:05:05.766 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:05:05.766 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:05:05.766 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:05:05.766 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836512 -lt 21474836512 2026-02-13T02:05:05.767 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:05:05.767 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-343597383683 2026-02-13T02:05:05.767 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:05:05.767 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:05:05.767 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-343597383683 2026-02-13T02:05:05.767 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:05:05.767 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=343597383683 2026-02-13T02:05:05.767 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 343597383683' 2026-02-13T02:05:05.767 INFO:tasks.workunit.client.0.trial112.stderr:157743: waiting osd.1 seq 343597383683 2026-02-13T02:05:05.767 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:05:05.767 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 343597383683 -lt 343597383683 2026-02-13T02:05:05.767 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:05:05.767 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-296352743434 2026-02-13T02:05:05.768 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:05:05.768 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:05:05.768 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-296352743434 2026-02-13T02:05:05.768 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:05:05.768 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=296352743434 2026-02-13T02:05:05.768 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 296352743434' 2026-02-13T02:05:05.768 INFO:tasks.workunit.client.0.trial112.stderr:157743: waiting osd.2 seq 296352743434 2026-02-13T02:05:05.768 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:05:06.430 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 2helpers.sh:2267: flush_pg_stats: echo 3-343597383682 2026-02-13T02:05:06.430 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:05:06.430 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:05:06.430 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-343597383682 2026-02-13T02:05:06.430 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:05:06.430 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=343597383682 2026-02-13T02:05:06.430 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 343597383682' 2026-02-13T02:05:06.431 INFO:tasks.workunit.client.0.trial112.stderr:157745: waiting osd.3 seq 343597383682 2026-02-13T02:05:06.431 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:05:06.431 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 343597383683 -lt 343597383682 2026-02-13T02:05:06.431 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:05:06.431 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:05:06.431 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:05:06.431 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:05:06.431 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:05:06.431 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:05:06.431 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:05:06.431 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:05:06.431 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:05:06.432 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:05:06.432 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:05:06.432 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:05:06.432 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:05:06.432 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:05:06.432 INFO:tasks.workunit.client.0.trial112.stderr:157745: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:05:06.432 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:05:06.432 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:05:06.432 INFO:tasks.workunit.client.0.trial112.stderr:157745: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:05:06.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-13T02:05:06.490 INFO:tasks.workunit.client.0.trial112.stderr:96352743434 -lt 296352743434 2026-02-13T02:05:06.490 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:05:06.490 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-343597383683 2026-02-13T02:05:06.490 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:05:06.490 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:05:06.490 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-343597383683 2026-02-13T02:05:06.490 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:05:06.490 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=343597383683 2026-02-13T02:05:06.490 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 343597383683' 2026-02-13T02:05:06.490 INFO:tasks.workunit.client.0.trial112.stderr:157743: waiting osd.3 seq 343597383683 2026-02-13T02:05:06.490 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:05:06.490 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 343597383683 -lt 343597383683 2026-02-13T02:05:06.490 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:05:06.490 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:05:06.490 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:05:06.490 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:05:06.490 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:05:06.490 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:05:06.490 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:05:06.490 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:05:06.490 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:05:06.490 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:05:06.490 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:05:06.490 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:05:06.490 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:05:06.490 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:05:06.491 INFO:tasks.workunit.client.0.trial112.stderr:157743: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:05:06.491 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:05:06.491 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:05:06.491 INFO:tasks.workunit.client.0.trial112.stderr:157743: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:05:06.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-13T02:05:06.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-13T02:05:06.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 157742 2026-02-13T02:05:06.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-13T02:05:06.492 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-13T02:05:06.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-13T02:05:06.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:272: corrupt_and_repair_two: return_code=0 2026-02-13T02:05:06.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-13T02:05:06.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:275: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-13T02:05:06.510 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:276: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-13T02:05:06.512 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-13T02:05:06.512 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:05:06.512 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:05:06.512 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:05:06.512 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:05:06.512 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:05:06.512 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:05:06.512 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:05:06.513 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:05:06.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:05:06.623 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:05:06.624 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:05:06.625 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:05:06.626 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:05:06.626 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:05:06.626 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:05:06.627 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:05:06.627 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:05:06.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:05:06.628 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:05:06.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:05:06.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:05:06.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:05:06.655 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:05:06.655 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:05:06.655 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:05:06.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:05:06.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:05:06.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:05:06.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-13T02:05:06.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-13T02:05:06.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-13T02:05:06.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-13T02:05:06.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:05:06.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:05:06.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:05:06.658 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:05:06.658 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:05:06.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:05:06.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:05:06.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:05:06.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:05:06.661 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:05:06.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:05:06.663 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:05:06.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:05:06.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:05:06.664 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:05:06.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:05:06.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:05:06.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:05:06.666 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:05:06.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:05:06.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:05:06.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:05:06.670 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:05:06.670 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:05:06.671 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:05:06.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:05:06.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:05:06.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:05:06.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-13T02:05:06.673 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T02:05:06.674 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:05:06.674 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:05:06.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.35207 2026-02-13T02:05:06.676 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T02:05:06.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-13T02:05:06.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T02:05:06.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-13T02:05:06.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_lrc_appends td/osd-scrub-repair 2026-02-13T02:05:06.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:901: TEST_corrupt_and_repair_lrc_appends: corrupt_and_repair_lrc td/osd-scrub-repair false 2026-02-13T02:05:06.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:884: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2026-02-13T02:05:06.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:885: corrupt_and_repair_lrc: local allow_overwrites=false 2026-02-13T02:05:06.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:886: corrupt_and_repair_lrc: local poolname=ecpool 2026-02-13T02:05:06.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:888: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2026-02-13T02:05:06.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-13T02:05:06.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-13T02:05:06.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-13T02:05:06.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-13T02:05:06.678 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-13T02:05:06.678 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-13T02:05:06.698 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-13T02:05:06.698 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:05:06.698 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:05:06.699 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:05:06.699 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:05:06.699 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:05:06.699 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:05:06.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-13T02:05:06.729 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-13T02:05:06.730 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-13T02:05:06.730 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:05:06.730 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:05:06.730 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-13T02:05:06.730 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-13T02:05:06.731 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:05:06.731 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:05:06.731 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:05:06.731 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:05:06.731 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:05:06.731 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:05:06.732 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:05:06.732 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:05:06.732 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get fsid 2026-02-13T02:05:06.804 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-13T02:05:06.804 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:05:06.804 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:05:06.804 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-13T02:05:06.804 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-13T02:05:06.805 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:05:06.805 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:05:06.805 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:05:06.805 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:05:06.805 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:05:06.805 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:05:06.806 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:05:06.806 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:05:06.806 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get mon_host 2026-02-13T02:05:06.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:889: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2026-02-13T02:05:06.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-13T02:05:06.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-13T02:05:06.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-13T02:05:06.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-13T02:05:06.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-13T02:05:06.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-13T02:05:06.953 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-13T02:05:06.954 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:05:06.954 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:05:06.954 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:05:06.954 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:05:06.954 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:05:06.955 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:05:06.955 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:05:06.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:05:06.975 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: seq 0 9 2026-02-13T02:05:06.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-13T02:05:06.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 0 2026-02-13T02:05:06.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:05:06.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:05:06.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-13T02:05:06.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:05:06.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:05:06.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:05:06.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:05:06.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:05:06.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:05:06.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:05:06.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:05:06.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:05:06.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:05:06.980 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:05:06.980 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:05:06.980 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:05:06.980 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:05:06.980 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:05:06.980 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:05:06.980 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:05:06.980 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:05:06.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:05:06.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:05:06.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:05:06.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:05:06.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:05:06.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:05:06.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:05:06.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:05:06.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:05:06.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:05:06.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:05:06.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:05:06.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:05:06.983 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:05:06.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=327f06ef-bb40-4d03-be2c-66bc35c52476 2026-02-13T02:05:06.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 327f06ef-bb40-4d03-be2c-66bc35c52476' 2026-02-13T02:05:06.984 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 327f06ef-bb40-4d03-be2c-66bc35c52476 2026-02-13T02:05:06.984 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:05:06.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDSho5pxUfUOhAADcWKGBbLdLtHTmNCWhMhQQ== 2026-02-13T02:05:06.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDSho5pxUfUOhAADcWKGBbLdLtHTmNCWhMhQQ=="}' 2026-02-13T02:05:06.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 327f06ef-bb40-4d03-be2c-66bc35c52476 -i td/osd-scrub-repair/0/new.json 2026-02-13T02:05:07.089 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:05:07.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-13T02:05:07.099 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDSho5pxUfUOhAADcWKGBbLdLtHTmNCWhMhQQ== --osd-uuid 327f06ef-bb40-4d03-be2c-66bc35c52476 2026-02-13T02:05:07.115 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:07.105+0000 7f8367d52a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:07.117 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:07.105+0000 7f8367d52a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:07.117 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:07.109+0000 7f8367d52a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:07.118 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:07.109+0000 7f8367d52a40 -1 bdev(0x564bed632c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-13T02:05:07.118 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:07.109+0000 7f8367d52a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-13T02:05:07.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-13T02:05:07.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:05:07.838 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T02:05:07.839 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-13T02:05:07.839 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:05:07.942 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T02:05:07.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-13T02:05:07.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:05:07.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:05:07.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:05:07.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:05:07.966 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:07.957+0000 7fb1c408ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:07.968 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:07.957+0000 7fb1c408ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:07.969 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:07.961+0000 7fb1c408ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:08.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-13T02:05:08.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:05:08.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:05:08.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:05:08.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:05:08.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:05:08.035 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:05:08.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:05:08.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:05:08.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:05:08.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:05:08.454 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:08.445+0000 7fb1c408ca40 -1 Falling back to public interface 2026-02-13T02:05:09.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:05:09.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:05:09.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:05:09.127 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:05:09.128 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:05:09.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:05:09.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:05:09.475 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:09.465+0000 7fb1c408ca40 -1 osd.0 0 log_to_monitors true 2026-02-13T02:05:10.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:05:10.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:05:10.240 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:05:10.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:05:10.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:05:10.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:05:10.401 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/335584748,v1:127.0.0.1:6801/335584748] [v2:127.0.0.1:6802/335584748,v1:127.0.0.1:6803/335584748] exists,up 327f06ef-bb40-4d03-be2c-66bc35c52476 2026-02-13T02:05:10.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:05:10.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:05:10.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:05:10.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-13T02:05:10.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 1 2026-02-13T02:05:10.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:05:10.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:05:10.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-13T02:05:10.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:05:10.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:05:10.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:05:10.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:05:10.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:05:10.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:05:10.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:05:10.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:05:10.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:05:10.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:05:10.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:05:10.406 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:05:10.406 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:05:10.406 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:05:10.406 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:05:10.406 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:05:10.406 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:05:10.406 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:05:10.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:05:10.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:05:10.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:05:10.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:05:10.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:05:10.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:05:10.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:05:10.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:05:10.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:05:10.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:05:10.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:05:10.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:05:10.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:05:10.408 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:05:10.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=001480b5-a9e3-4f60-8a79-c27a9fd48356 2026-02-13T02:05:10.408 INFO:tasks.workunit.client.0.trial112.stdout:add osd1 001480b5-a9e3-4f60-8a79-c27a9fd48356 2026-02-13T02:05:10.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 001480b5-a9e3-4f60-8a79-c27a9fd48356' 2026-02-13T02:05:10.409 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:05:10.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDWho5pRziMGBAAhnaVs+oziKFKCkQAy1s4MQ== 2026-02-13T02:05:10.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDWho5pRziMGBAAhnaVs+oziKFKCkQAy1s4MQ=="}' 2026-02-13T02:05:10.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 001480b5-a9e3-4f60-8a79-c27a9fd48356 -i td/osd-scrub-repair/1/new.json 2026-02-13T02:05:10.587 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:05:10.598 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-13T02:05:10.599 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDWho5pRziMGBAAhnaVs+oziKFKCkQAy1s4MQ== --osd-uuid 001480b5-a9e3-4f60-8a79-c27a9fd48356 2026-02-13T02:05:10.626 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:10.617+0000 7fdb86291a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:10.627 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:10.617+0000 7fdb86291a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:10.628 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:10.617+0000 7fdb86291a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:10.629 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:10.617+0000 7fdb86291a40 -1 bdev(0x56245c8c4c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-13T02:05:10.629 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:10.617+0000 7fdb86291a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-13T02:05:11.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-13T02:05:11.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:05:11.258 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-13T02:05:11.258 INFO:tasks.workunit.client.0.trial112.stdout:adding osd1 key to auth repository 2026-02-13T02:05:11.258 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:05:11.465 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T02:05:11.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-13T02:05:11.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:05:11.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:05:11.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:05:11.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:05:11.490 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:11.481+0000 7ff97f528a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:11.492 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:11.481+0000 7ff97f528a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:11.493 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:11.481+0000 7ff97f528a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:11.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-13T02:05:11.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:05:11.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:05:11.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:05:11.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:05:11.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:05:11.631 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:05:11.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:05:11.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:05:11.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:05:11.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:05:11.859 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:11.849+0000 7ff97f528a40 -1 Falling back to public interface 2026-02-13T02:05:12.423 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:12.413+0000 7ff97f528a40 -1 osd.1 0 log_to_monitors true 2026-02-13T02:05:12.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:05:12.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:05:12.793 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:05:12.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:05:12.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:05:12.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:05:12.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:05:13.950 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:05:13.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:05:13.951 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:05:13.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:05:13.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:05:13.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:05:14.108 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1103259071,v1:127.0.0.1:6811/1103259071] [v2:127.0.0.1:6812/1103259071,v1:127.0.0.1:6813/1103259071] exists,up 001480b5-a9e3-4f60-8a79-c27a9fd48356 2026-02-13T02:05:14.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:05:14.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:05:14.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:05:14.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-13T02:05:14.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 2 2026-02-13T02:05:14.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:05:14.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:05:14.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-13T02:05:14.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:05:14.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-13T02:05:14.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:05:14.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:05:14.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:05:14.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:05:14.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-13T02:05:14.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-13T02:05:14.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:05:14.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:05:14.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:05:14.113 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:05:14.113 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:05:14.113 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:05:14.113 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:05:14.113 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:05:14.113 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:05:14.113 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:05:14.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:05:14.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:05:14.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:05:14.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:05:14.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:05:14.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:05:14.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:05:14.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:05:14.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:05:14.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:05:14.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:05:14.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:05:14.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-13T02:05:14.115 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:05:14.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f363539c-c72d-47af-871b-a5e53997e74f 2026-02-13T02:05:14.115 INFO:tasks.workunit.client.0.trial112.stdout:add osd2 f363539c-c72d-47af-871b-a5e53997e74f 2026-02-13T02:05:14.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 f363539c-c72d-47af-871b-a5e53997e74f' 2026-02-13T02:05:14.116 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:05:14.125 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDaho5pWPACBxAAhfHFLS4eNLgxdZpVOdX4pQ== 2026-02-13T02:05:14.125 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDaho5pWPACBxAAhfHFLS4eNLgxdZpVOdX4pQ=="}' 2026-02-13T02:05:14.125 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f363539c-c72d-47af-871b-a5e53997e74f -i td/osd-scrub-repair/2/new.json 2026-02-13T02:05:14.289 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:05:14.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-13T02:05:14.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDaho5pWPACBxAAhfHFLS4eNLgxdZpVOdX4pQ== --osd-uuid f363539c-c72d-47af-871b-a5e53997e74f 2026-02-13T02:05:14.315 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:14.305+0000 7f5afb4fda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:14.316 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:14.305+0000 7f5afb4fda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:14.318 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:14.309+0000 7f5afb4fda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:14.318 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:14.309+0000 7f5afb4fda40 -1 bdev(0x556576958c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-13T02:05:14.318 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:14.309+0000 7f5afb4fda40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-13T02:05:14.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-13T02:05:14.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:05:14.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-13T02:05:14.965 INFO:tasks.workunit.client.0.trial112.stdout:adding osd2 key to auth repository 2026-02-13T02:05:14.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:05:15.170 INFO:tasks.workunit.client.0.trial112.stdout:start osd.2 2026-02-13T02:05:15.170 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-13T02:05:15.170 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:05:15.170 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:05:15.170 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:05:15.170 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:05:15.194 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:15.185+0000 7fdb5626aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:15.195 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:15.185+0000 7fdb5626aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:15.196 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:15.185+0000 7fdb5626aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:15.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-13T02:05:15.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:05:15.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T02:05:15.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:05:15.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:05:15.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:05:15.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:05:15.339 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:05:15.339 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:05:15.339 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:05:15.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:05:15.567 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:15.557+0000 7fdb5626aa40 -1 Falling back to public interface 2026-02-13T02:05:16.197 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:16.185+0000 7fdb5626aa40 -1 osd.2 0 log_to_monitors true 2026-02-13T02:05:16.502 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:05:16.502 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:05:16.503 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:05:16.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:05:16.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:05:16.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:05:16.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:05:17.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:05:17.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:05:17.666 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:05:17.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:05:17.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:05:17.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:05:17.820 INFO:tasks.workunit.client.0.trial112.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3494222359,v1:127.0.0.1:6819/3494222359] [v2:127.0.0.1:6820/3494222359,v1:127.0.0.1:6821/3494222359] exists,up f363539c-c72d-47af-871b-a5e53997e74f 2026-02-13T02:05:17.823 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:05:17.823 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:05:17.823 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:05:17.823 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-13T02:05:17.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 3 2026-02-13T02:05:17.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:05:17.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:05:17.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-13T02:05:17.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:05:17.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:05:17.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:05:17.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:05:17.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:05:17.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:05:17.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-13T02:05:17.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-13T02:05:17.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:05:17.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:05:17.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:05:17.825 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:05:17.825 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:05:17.825 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:05:17.825 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:05:17.825 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:05:17.825 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:05:17.825 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:05:17.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:05:17.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:05:17.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:05:17.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:05:17.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:05:17.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:05:17.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:05:17.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:05:17.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:05:17.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:05:17.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:05:17.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:05:17.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/3 2026-02-13T02:05:17.827 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:05:17.827 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ad30d011-cd62-4310-8456-c2d78faa8a93 2026-02-13T02:05:17.827 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 ad30d011-cd62-4310-8456-c2d78faa8a93' 2026-02-13T02:05:17.828 INFO:tasks.workunit.client.0.trial112.stdout:add osd3 ad30d011-cd62-4310-8456-c2d78faa8a93 2026-02-13T02:05:17.828 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:05:17.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDdho5pnvx0MRAAic8wiOKGH7yC8SsjRCbO2Q== 2026-02-13T02:05:17.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDdho5pnvx0MRAAic8wiOKGH7yC8SsjRCbO2Q=="}' 2026-02-13T02:05:17.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ad30d011-cd62-4310-8456-c2d78faa8a93 -i td/osd-scrub-repair/3/new.json 2026-02-13T02:05:17.997 INFO:tasks.workunit.client.0.trial112.stdout:3 2026-02-13T02:05:18.006 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/3/new.json 2026-02-13T02:05:18.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDdho5pnvx0MRAAic8wiOKGH7yC8SsjRCbO2Q== --osd-uuid ad30d011-cd62-4310-8456-c2d78faa8a93 2026-02-13T02:05:18.023 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:18.013+0000 7fd3c6abaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:18.025 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:18.013+0000 7fd3c6abaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:18.026 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:18.017+0000 7fd3c6abaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:18.027 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:18.017+0000 7fd3c6abaa40 -1 bdev(0x55a5e12f0c00 td/osd-scrub-repair/3/block) open stat got: (1) Operation not permitted 2026-02-13T02:05:18.027 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:18.017+0000 7fd3c6abaa40 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2026-02-13T02:05:18.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2026-02-13T02:05:18.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:05:18.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-13T02:05:18.673 INFO:tasks.workunit.client.0.trial112.stdout:adding osd3 key to auth repository 2026-02-13T02:05:18.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:05:18.867 INFO:tasks.workunit.client.0.trial112.stdout:start osd.3 2026-02-13T02:05:18.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-13T02:05:18.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:05:18.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:05:18.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:05:18.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:05:18.888 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:18.877+0000 7fac1310ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:18.890 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:18.881+0000 7fac1310ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:18.891 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:18.881+0000 7fac1310ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:19.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-13T02:05:19.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:05:19.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-13T02:05:19.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:05:19.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:05:19.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:05:19.031 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:05:19.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:05:19.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:05:19.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:05:19.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:05:19.315 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:19.305+0000 7fac1310ba40 -1 Falling back to public interface 2026-02-13T02:05:19.933 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:19.921+0000 7fac1310ba40 -1 osd.3 0 log_to_monitors true 2026-02-13T02:05:20.196 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:05:20.196 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:05:20.196 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:05:20.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:05:20.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:05:20.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:05:20.357 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:05:21.358 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:05:21.359 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:05:21.359 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:05:21.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:05:21.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:05:21.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:05:21.513 INFO:tasks.workunit.client.0.trial112.stdout:osd.3 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1071235144,v1:127.0.0.1:6827/1071235144] [v2:127.0.0.1:6828/1071235144,v1:127.0.0.1:6829/1071235144] exists,up ad30d011-cd62-4310-8456-c2d78faa8a93 2026-02-13T02:05:21.516 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:05:21.516 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:05:21.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:05:21.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-13T02:05:21.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 4 2026-02-13T02:05:21.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:05:21.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:05:21.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2026-02-13T02:05:21.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:05:21.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/4 2026-02-13T02:05:21.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:05:21.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:05:21.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:05:21.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:05:21.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2026-02-13T02:05:21.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2026-02-13T02:05:21.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:05:21.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:05:21.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:05:21.518 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:05:21.518 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:05:21.518 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:05:21.518 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:05:21.518 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:05:21.518 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:05:21.518 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:05:21.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:05:21.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:05:21.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:05:21.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:05:21.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:05:21.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:05:21.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:05:21.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:05:21.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:05:21.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:05:21.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:05:21.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:05:21.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/4 2026-02-13T02:05:21.520 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:05:21.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=fd8ee52b-f82e-4fc7-9462-b37851051358 2026-02-13T02:05:21.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 fd8ee52b-f82e-4fc7-9462-b37851051358' 2026-02-13T02:05:21.521 INFO:tasks.workunit.client.0.trial112.stdout:add osd4 fd8ee52b-f82e-4fc7-9462-b37851051358 2026-02-13T02:05:21.521 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:05:21.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDhho5pqcc3HxAARSqBz9xJR3LckNbHs2y5QA== 2026-02-13T02:05:21.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDhho5pqcc3HxAARSqBz9xJR3LckNbHs2y5QA=="}' 2026-02-13T02:05:21.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new fd8ee52b-f82e-4fc7-9462-b37851051358 -i td/osd-scrub-repair/4/new.json 2026-02-13T02:05:21.696 INFO:tasks.workunit.client.0.trial112.stdout:4 2026-02-13T02:05:21.707 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/4/new.json 2026-02-13T02:05:21.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDhho5pqcc3HxAARSqBz9xJR3LckNbHs2y5QA== --osd-uuid fd8ee52b-f82e-4fc7-9462-b37851051358 2026-02-13T02:05:21.731 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:21.721+0000 7f6069c70a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:21.733 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:21.721+0000 7f6069c70a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:21.734 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:21.725+0000 7f6069c70a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:21.734 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:21.725+0000 7f6069c70a40 -1 bdev(0x55cb7d690c00 td/osd-scrub-repair/4/block) open stat got: (1) Operation not permitted 2026-02-13T02:05:21.735 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:21.725+0000 7f6069c70a40 -1 bluestore(td/osd-scrub-repair/4) _read_fsid unparsable uuid 2026-02-13T02:05:22.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/4/keyring 2026-02-13T02:05:22.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:05:22.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2026-02-13T02:05:22.374 INFO:tasks.workunit.client.0.trial112.stdout:adding osd4 key to auth repository 2026-02-13T02:05:22.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:05:22.582 INFO:tasks.workunit.client.0.trial112.stdout:start osd.4 2026-02-13T02:05:22.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2026-02-13T02:05:22.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:05:22.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:05:22.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:05:22.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:05:22.598 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:22.589+0000 7f0ba34d4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:22.600 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:22.589+0000 7f0ba34d4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:22.601 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:22.589+0000 7f0ba34d4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:22.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2026-02-13T02:05:22.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:05:22.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2026-02-13T02:05:22.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:05:22.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:05:22.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:05:22.744 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:05:22.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:05:22.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:05:22.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-13T02:05:22.899 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:05:23.027 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:23.017+0000 7f0ba34d4a40 -1 Falling back to public interface 2026-02-13T02:05:23.634 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:23.625+0000 7f0ba34d4a40 -1 osd.4 0 log_to_monitors true 2026-02-13T02:05:23.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:05:23.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:05:23.901 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:05:23.901 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:05:23.901 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:05:23.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-13T02:05:24.064 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:05:25.066 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:05:25.066 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:05:25.067 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:05:25.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:05:25.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:05:25.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-13T02:05:25.230 INFO:tasks.workunit.client.0.trial112.stdout:osd.4 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/1449768323,v1:127.0.0.1:6835/1449768323] [v2:127.0.0.1:6836/1449768323,v1:127.0.0.1:6837/1449768323] exists,up fd8ee52b-f82e-4fc7-9462-b37851051358 2026-02-13T02:05:25.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:05:25.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:05:25.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:05:25.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-13T02:05:25.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 5 2026-02-13T02:05:25.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:05:25.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:05:25.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2026-02-13T02:05:25.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:05:25.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/5 2026-02-13T02:05:25.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:05:25.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:05:25.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:05:25.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:05:25.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2026-02-13T02:05:25.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2026-02-13T02:05:25.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:05:25.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:05:25.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:05:25.235 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:05:25.235 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:05:25.236 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:05:25.236 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:05:25.236 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:05:25.236 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:05:25.236 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:05:25.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:05:25.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:05:25.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:05:25.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:05:25.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:05:25.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:05:25.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:05:25.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:05:25.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:05:25.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:05:25.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:05:25.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:05:25.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/5 2026-02-13T02:05:25.237 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:05:25.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=670b1f5b-3687-4794-8a8a-767a3a2770e7 2026-02-13T02:05:25.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 670b1f5b-3687-4794-8a8a-767a3a2770e7' 2026-02-13T02:05:25.238 INFO:tasks.workunit.client.0.trial112.stdout:add osd5 670b1f5b-3687-4794-8a8a-767a3a2770e7 2026-02-13T02:05:25.238 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:05:25.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDlho5pCT9ODhAAJUSxgxbDmw+A9gi7oS4f+w== 2026-02-13T02:05:25.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDlho5pCT9ODhAAJUSxgxbDmw+A9gi7oS4f+w=="}' 2026-02-13T02:05:25.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 670b1f5b-3687-4794-8a8a-767a3a2770e7 -i td/osd-scrub-repair/5/new.json 2026-02-13T02:05:25.411 INFO:tasks.workunit.client.0.trial112.stdout:5 2026-02-13T02:05:25.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/5/new.json 2026-02-13T02:05:25.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDlho5pCT9ODhAAJUSxgxbDmw+A9gi7oS4f+w== --osd-uuid 670b1f5b-3687-4794-8a8a-767a3a2770e7 2026-02-13T02:05:25.438 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:25.429+0000 7f1ac00dba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:25.439 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:25.429+0000 7f1ac00dba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:25.440 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:25.429+0000 7f1ac00dba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:25.441 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:25.429+0000 7f1ac00dba40 -1 bdev(0x560b988aec00 td/osd-scrub-repair/5/block) open stat got: (1) Operation not permitted 2026-02-13T02:05:25.441 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:25.429+0000 7f1ac00dba40 -1 bluestore(td/osd-scrub-repair/5) _read_fsid unparsable uuid 2026-02-13T02:05:26.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/5/keyring 2026-02-13T02:05:26.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:05:26.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2026-02-13T02:05:26.077 INFO:tasks.workunit.client.0.trial112.stdout:adding osd5 key to auth repository 2026-02-13T02:05:26.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:05:26.280 INFO:tasks.workunit.client.0.trial112.stdout:start osd.5 2026-02-13T02:05:26.280 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2026-02-13T02:05:26.280 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:05:26.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:05:26.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:05:26.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:05:26.304 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:26.293+0000 7f62b9491a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:26.306 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:26.297+0000 7f62b9491a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:26.307 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:26.297+0000 7f62b9491a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:26.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2026-02-13T02:05:26.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:05:26.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-13T02:05:26.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:05:26.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:05:26.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:05:26.444 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:05:26.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:05:26.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:05:26.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-13T02:05:26.600 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:05:26.675 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:26.665+0000 7f62b9491a40 -1 Falling back to public interface 2026-02-13T02:05:27.283 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:27.273+0000 7f62b9491a40 -1 osd.5 0 log_to_monitors true 2026-02-13T02:05:27.601 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:05:27.601 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:05:27.602 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:05:27.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:05:27.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:05:27.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-13T02:05:27.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:05:28.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:05:28.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:05:28.756 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:05:28.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:05:28.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:05:28.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-13T02:05:28.912 INFO:tasks.workunit.client.0.trial112.stdout:osd.5 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3772570450,v1:127.0.0.1:6843/3772570450] [v2:127.0.0.1:6844/3772570450,v1:127.0.0.1:6845/3772570450] exists,up 670b1f5b-3687-4794-8a8a-767a3a2770e7 2026-02-13T02:05:28.913 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:05:28.913 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:05:28.913 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:05:28.913 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-13T02:05:28.913 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 6 2026-02-13T02:05:28.913 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:05:28.913 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:05:28.913 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2026-02-13T02:05:28.913 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:05:28.913 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/6 2026-02-13T02:05:28.913 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:05:28.913 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:05:28.913 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:05:28.913 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:05:28.913 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2026-02-13T02:05:28.913 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2026-02-13T02:05:28.913 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:05:28.913 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:05:28.914 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:05:28.914 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:05:28.914 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:05:28.914 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:05:28.914 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:05:28.914 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:05:28.914 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:05:28.914 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:05:28.915 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:05:28.915 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:05:28.915 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:05:28.915 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:05:28.915 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:05:28.915 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:05:28.915 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:05:28.915 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:05:28.915 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:05:28.915 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:05:28.915 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:05:28.915 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:05:28.915 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/6 2026-02-13T02:05:28.916 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:05:28.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4b405227-52f9-48b0-8a16-d1b2414264ea 2026-02-13T02:05:28.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 4b405227-52f9-48b0-8a16-d1b2414264ea' 2026-02-13T02:05:28.917 INFO:tasks.workunit.client.0.trial112.stdout:add osd6 4b405227-52f9-48b0-8a16-d1b2414264ea 2026-02-13T02:05:28.918 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:05:28.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDoho5pweoLNxAArkvy0S/IE6yJgyG/k79nGA== 2026-02-13T02:05:28.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDoho5pweoLNxAArkvy0S/IE6yJgyG/k79nGA=="}' 2026-02-13T02:05:28.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4b405227-52f9-48b0-8a16-d1b2414264ea -i td/osd-scrub-repair/6/new.json 2026-02-13T02:05:29.091 INFO:tasks.workunit.client.0.trial112.stdout:6 2026-02-13T02:05:29.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/6/new.json 2026-02-13T02:05:29.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDoho5pweoLNxAArkvy0S/IE6yJgyG/k79nGA== --osd-uuid 4b405227-52f9-48b0-8a16-d1b2414264ea 2026-02-13T02:05:29.119 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:29.109+0000 7f8d819cda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:29.120 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:29.109+0000 7f8d819cda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:29.121 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:29.113+0000 7f8d819cda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:29.122 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:29.113+0000 7f8d819cda40 -1 bdev(0x561a46246c00 td/osd-scrub-repair/6/block) open stat got: (1) Operation not permitted 2026-02-13T02:05:29.122 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:29.113+0000 7f8d819cda40 -1 bluestore(td/osd-scrub-repair/6) _read_fsid unparsable uuid 2026-02-13T02:05:29.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/6/keyring 2026-02-13T02:05:29.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:05:29.741 INFO:tasks.workunit.client.0.trial112.stdout:adding osd6 key to auth repository 2026-02-13T02:05:29.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2026-02-13T02:05:29.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:05:29.945 INFO:tasks.workunit.client.0.trial112.stdout:start osd.6 2026-02-13T02:05:29.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2026-02-13T02:05:29.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:05:29.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:05:29.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:05:29.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:05:29.968 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:29.957+0000 7efe51a33a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:29.970 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:29.961+0000 7efe51a33a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:29.971 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:29.961+0000 7efe51a33a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:30.105 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2026-02-13T02:05:30.105 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:05:30.105 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2026-02-13T02:05:30.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:05:30.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:05:30.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:05:30.106 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:05:30.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:05:30.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:05:30.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-13T02:05:30.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:05:30.382 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:30.373+0000 7efe51a33a40 -1 Falling back to public interface 2026-02-13T02:05:31.047 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:31.037+0000 7efe51a33a40 -1 osd.6 0 log_to_monitors true 2026-02-13T02:05:31.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:05:31.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:05:31.265 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:05:31.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:05:31.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:05:31.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-13T02:05:31.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:05:32.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:05:32.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:05:32.424 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:05:32.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:05:32.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:05:32.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-13T02:05:32.579 INFO:tasks.workunit.client.0.trial112.stdout:osd.6 up in weight 1 up_from 32 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/2189445897,v1:127.0.0.1:6851/2189445897] [v2:127.0.0.1:6852/2189445897,v1:127.0.0.1:6853/2189445897] exists,up 4b405227-52f9-48b0-8a16-d1b2414264ea 2026-02-13T02:05:32.580 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:05:32.580 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:05:32.580 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:05:32.580 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-13T02:05:32.580 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 7 2026-02-13T02:05:32.580 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:05:32.580 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:05:32.580 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=7 2026-02-13T02:05:32.580 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:05:32.580 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/7 2026-02-13T02:05:32.580 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:05:32.581 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:05:32.581 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:05:32.581 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:05:32.581 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2026-02-13T02:05:32.581 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2026-02-13T02:05:32.581 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:05:32.581 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:05:32.581 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:05:32.581 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:05:32.581 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:05:32.581 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:05:32.581 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:05:32.581 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:05:32.581 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:05:32.581 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:05:32.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:05:32.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:05:32.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:05:32.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:05:32.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:05:32.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:05:32.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:05:32.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:05:32.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:05:32.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:05:32.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:05:32.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:05:32.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/7 2026-02-13T02:05:32.583 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:05:32.584 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f1900e8f-da04-4cd7-b9db-e176181ead2e 2026-02-13T02:05:32.584 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd7 f1900e8f-da04-4cd7-b9db-e176181ead2e' 2026-02-13T02:05:32.584 INFO:tasks.workunit.client.0.trial112.stdout:add osd7 f1900e8f-da04-4cd7-b9db-e176181ead2e 2026-02-13T02:05:32.584 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:05:32.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDsho5phZlyIxAAB0Hp/Ytok6bGjVE7i6KYQQ== 2026-02-13T02:05:32.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDsho5phZlyIxAAB0Hp/Ytok6bGjVE7i6KYQQ=="}' 2026-02-13T02:05:32.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f1900e8f-da04-4cd7-b9db-e176181ead2e -i td/osd-scrub-repair/7/new.json 2026-02-13T02:05:32.756 INFO:tasks.workunit.client.0.trial112.stdout:7 2026-02-13T02:05:32.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/7/new.json 2026-02-13T02:05:32.766 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 7 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDsho5phZlyIxAAB0Hp/Ytok6bGjVE7i6KYQQ== --osd-uuid f1900e8f-da04-4cd7-b9db-e176181ead2e 2026-02-13T02:05:32.783 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:32.773+0000 7f19a6f0ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:32.784 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:32.773+0000 7f19a6f0ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:32.785 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:32.777+0000 7f19a6f0ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:32.786 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:32.777+0000 7f19a6f0ea40 -1 bdev(0x560f2fe74c00 td/osd-scrub-repair/7/block) open stat got: (1) Operation not permitted 2026-02-13T02:05:32.786 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:32.777+0000 7f19a6f0ea40 -1 bluestore(td/osd-scrub-repair/7) _read_fsid unparsable uuid 2026-02-13T02:05:33.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/7/keyring 2026-02-13T02:05:33.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:05:33.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd7 key to auth repository 2026-02-13T02:05:33.438 INFO:tasks.workunit.client.0.trial112.stdout:adding osd7 key to auth repository 2026-02-13T02:05:33.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:05:33.640 INFO:tasks.workunit.client.0.trial112.stdout:start osd.7 2026-02-13T02:05:33.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.7 2026-02-13T02:05:33.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 7 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:05:33.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:05:33.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:05:33.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:05:33.665 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:33.653+0000 7f33a9109a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:33.666 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:33.657+0000 7f33a9109a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:33.668 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:33.657+0000 7f33a9109a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:33.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 7 2026-02-13T02:05:33.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:05:33.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=7 2026-02-13T02:05:33.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:05:33.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:05:33.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:05:33.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:05:33.803 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:05:33.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:05:33.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2026-02-13T02:05:33.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:05:34.030 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:34.021+0000 7f33a9109a40 -1 Falling back to public interface 2026-02-13T02:05:34.625 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:34.613+0000 7f33a9109a40 -1 osd.7 0 log_to_monitors true 2026-02-13T02:05:34.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:05:34.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:05:34.967 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:05:34.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:05:34.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:05:34.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2026-02-13T02:05:35.129 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:05:36.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:05:36.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:05:36.131 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:05:36.132 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:05:36.132 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:05:36.132 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2026-02-13T02:05:36.285 INFO:tasks.workunit.client.0.trial112.stdout:osd.7 up in weight 1 up_from 37 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/144668283,v1:127.0.0.1:6859/144668283] [v2:127.0.0.1:6860/144668283,v1:127.0.0.1:6861/144668283] exists,up f1900e8f-da04-4cd7-b9db-e176181ead2e 2026-02-13T02:05:36.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:05:36.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:05:36.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:05:36.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-13T02:05:36.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 8 2026-02-13T02:05:36.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:05:36.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:05:36.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=8 2026-02-13T02:05:36.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:05:36.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/8 2026-02-13T02:05:36.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:05:36.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:05:36.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:05:36.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:05:36.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2026-02-13T02:05:36.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2026-02-13T02:05:36.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:05:36.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:05:36.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:05:36.286 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:05:36.286 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:05:36.286 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:05:36.286 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:05:36.286 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:05:36.286 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:05:36.287 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:05:36.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:05:36.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:05:36.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:05:36.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:05:36.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:05:36.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:05:36.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:05:36.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:05:36.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:05:36.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:05:36.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:05:36.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:05:36.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/8 2026-02-13T02:05:36.288 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:05:36.290 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=62de9b77-51b7-4e98-9c9a-74e6f6b53412 2026-02-13T02:05:36.290 INFO:tasks.workunit.client.0.trial112.stdout:add osd8 62de9b77-51b7-4e98-9c9a-74e6f6b53412 2026-02-13T02:05:36.290 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd8 62de9b77-51b7-4e98-9c9a-74e6f6b53412' 2026-02-13T02:05:36.290 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:05:36.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDwho5pkimQERAAxKZduRCLVKLpadJrSfrgiA== 2026-02-13T02:05:36.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDwho5pkimQERAAxKZduRCLVKLpadJrSfrgiA=="}' 2026-02-13T02:05:36.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 62de9b77-51b7-4e98-9c9a-74e6f6b53412 -i td/osd-scrub-repair/8/new.json 2026-02-13T02:05:36.458 INFO:tasks.workunit.client.0.trial112.stdout:8 2026-02-13T02:05:36.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/8/new.json 2026-02-13T02:05:36.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 8 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDwho5pkimQERAAxKZduRCLVKLpadJrSfrgiA== --osd-uuid 62de9b77-51b7-4e98-9c9a-74e6f6b53412 2026-02-13T02:05:36.484 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:36.473+0000 7f3054388a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:36.485 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:36.477+0000 7f3054388a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:36.486 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:36.477+0000 7f3054388a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:36.486 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:36.477+0000 7f3054388a40 -1 bdev(0x55f1b331ac00 td/osd-scrub-repair/8/block) open stat got: (1) Operation not permitted 2026-02-13T02:05:36.487 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:36.477+0000 7f3054388a40 -1 bluestore(td/osd-scrub-repair/8) _read_fsid unparsable uuid 2026-02-13T02:05:37.099 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/8/keyring 2026-02-13T02:05:37.099 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:05:37.100 INFO:tasks.workunit.client.0.trial112.stdout:adding osd8 key to auth repository 2026-02-13T02:05:37.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd8 key to auth repository 2026-02-13T02:05:37.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:05:37.302 INFO:tasks.workunit.client.0.trial112.stdout:start osd.8 2026-02-13T02:05:37.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.8 2026-02-13T02:05:37.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 8 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:05:37.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:05:37.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:05:37.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:05:37.326 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:37.317+0000 7fc46ae3ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:37.328 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:37.317+0000 7fc46ae3ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:37.329 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:37.317+0000 7fc46ae3ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:37.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 8 2026-02-13T02:05:37.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:05:37.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=8 2026-02-13T02:05:37.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:05:37.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:05:37.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:05:37.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:05:37.468 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:05:37.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:05:37.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2026-02-13T02:05:37.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:05:37.719 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:37.709+0000 7fc46ae3ca40 -1 Falling back to public interface 2026-02-13T02:05:38.365 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:38.353+0000 7fc46ae3ca40 -1 osd.8 0 log_to_monitors true 2026-02-13T02:05:38.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:05:38.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:05:38.631 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:05:38.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:05:38.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:05:38.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2026-02-13T02:05:38.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:05:39.795 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:05:39.795 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:05:39.795 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:05:39.795 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:05:39.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:05:39.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2026-02-13T02:05:39.952 INFO:tasks.workunit.client.0.trial112.stdout:osd.8 up in weight 1 up_from 42 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/3160667913,v1:127.0.0.1:6867/3160667913] [v2:127.0.0.1:6868/3160667913,v1:127.0.0.1:6869/3160667913] exists,up 62de9b77-51b7-4e98-9c9a-74e6f6b53412 2026-02-13T02:05:39.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:05:39.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:05:39.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:05:39.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-13T02:05:39.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 9 2026-02-13T02:05:39.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:05:39.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:05:39.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=9 2026-02-13T02:05:39.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:05:39.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/9 2026-02-13T02:05:39.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:05:39.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:05:39.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:05:39.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:05:39.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2026-02-13T02:05:39.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2026-02-13T02:05:39.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:05:39.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:05:39.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:05:39.957 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:05:39.957 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:05:39.957 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:05:39.957 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:05:39.957 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:05:39.957 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:05:39.958 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:05:39.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:05:39.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:05:39.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:05:39.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:05:39.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:05:39.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:05:39.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:05:39.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:05:39.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:05:39.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:05:39.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:05:39.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:05:39.959 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/9 2026-02-13T02:05:39.959 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:05:39.959 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=31da8501-3453-451f-af71-20390baae5a0 2026-02-13T02:05:39.960 INFO:tasks.workunit.client.0.trial112.stdout:add osd9 31da8501-3453-451f-af71-20390baae5a0 2026-02-13T02:05:39.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd9 31da8501-3453-451f-af71-20390baae5a0' 2026-02-13T02:05:39.960 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:05:39.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDzho5pBTRSORAAdCylkkUGs3en3omn2ggdUA== 2026-02-13T02:05:39.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDzho5pBTRSORAAdCylkkUGs3en3omn2ggdUA=="}' 2026-02-13T02:05:39.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 31da8501-3453-451f-af71-20390baae5a0 -i td/osd-scrub-repair/9/new.json 2026-02-13T02:05:40.128 INFO:tasks.workunit.client.0.trial112.stdout:9 2026-02-13T02:05:40.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/9/new.json 2026-02-13T02:05:40.138 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 9 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDzho5pBTRSORAAdCylkkUGs3en3omn2ggdUA== --osd-uuid 31da8501-3453-451f-af71-20390baae5a0 2026-02-13T02:05:40.154 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:40.145+0000 7f9890462a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:40.156 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:40.145+0000 7f9890462a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:40.157 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:40.145+0000 7f9890462a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:40.158 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:40.149+0000 7f9890462a40 -1 bdev(0x557ae1978c00 td/osd-scrub-repair/9/block) open stat got: (1) Operation not permitted 2026-02-13T02:05:40.158 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:40.149+0000 7f9890462a40 -1 bluestore(td/osd-scrub-repair/9) _read_fsid unparsable uuid 2026-02-13T02:05:40.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/9/keyring 2026-02-13T02:05:40.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:05:40.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd9 key to auth repository 2026-02-13T02:05:40.782 INFO:tasks.workunit.client.0.trial112.stdout:adding osd9 key to auth repository 2026-02-13T02:05:40.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:05:40.980 INFO:tasks.workunit.client.0.trial112.stdout:start osd.9 2026-02-13T02:05:40.981 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.9 2026-02-13T02:05:40.981 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 9 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:05:40.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:05:40.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:05:40.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:05:41.005 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:40.993+0000 7f4c77fada40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:41.007 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:40.997+0000 7f4c77fada40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:41.008 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:40.997+0000 7f4c77fada40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:41.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 9 2026-02-13T02:05:41.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:05:41.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2026-02-13T02:05:41.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:05:41.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:05:41.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:05:41.141 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:05:41.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:05:41.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:05:41.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-13T02:05:41.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:05:41.435 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:41.425+0000 7f4c77fada40 -1 Falling back to public interface 2026-02-13T02:05:42.112 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:42.101+0000 7f4c77fada40 -1 osd.9 0 log_to_monitors true 2026-02-13T02:05:42.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:05:42.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:05:42.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:05:42.306 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:05:42.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:05:42.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-13T02:05:42.469 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:05:43.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:05:43.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:05:43.471 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:05:43.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:05:43.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:05:43.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-13T02:05:43.629 INFO:tasks.workunit.client.0.trial112.stdout:osd.9 up in weight 1 up_from 47 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/96640006,v1:127.0.0.1:6875/96640006] [v2:127.0.0.1:6876/96640006,v1:127.0.0.1:6877/96640006] exists,up 31da8501-3453-451f-af71-20390baae5a0 2026-02-13T02:05:43.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:05:43.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:05:43.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:05:43.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:893: corrupt_and_repair_lrc: create_rbd_pool 2026-02-13T02:05:43.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-13T02:05:43.782 INFO:tasks.workunit.client.0.trial112.stderr:pool 'rbd' does not exist 2026-02-13T02:05:43.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-13T02:05:43.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-13T02:05:44.004 INFO:tasks.workunit.client.0.trial112.stderr:pool 'rbd' created 2026-02-13T02:05:44.015 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:05:45.017 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-13T02:05:45.295 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: corrupt_and_repair_lrc: wait_for_clean 2026-02-13T02:05:45.295 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:05:45.295 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:05:45.295 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:05:45.295 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:05:45.296 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:05:45.296 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:05:45.296 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:05:45.297 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:05:45.297 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:05:45.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:05:45.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:05:45.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:05:45.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:05:45.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:05:45.409 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:05:45.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:05:45.574 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:05:45.574 INFO:tasks.workunit.client.0.trial112.stderr:2 2026-02-13T02:05:45.574 INFO:tasks.workunit.client.0.trial112.stderr:3 2026-02-13T02:05:45.574 INFO:tasks.workunit.client.0.trial112.stderr:4 2026-02-13T02:05:45.574 INFO:tasks.workunit.client.0.trial112.stderr:5 2026-02-13T02:05:45.574 INFO:tasks.workunit.client.0.trial112.stderr:6 2026-02-13T02:05:45.574 INFO:tasks.workunit.client.0.trial112.stderr:7 2026-02-13T02:05:45.574 INFO:tasks.workunit.client.0.trial112.stderr:8 2026-02-13T02:05:45.574 INFO:tasks.workunit.client.0.trial112.stderr:9' 2026-02-13T02:05:45.574 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:05:45.574 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:05:45.575 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:05:45.644 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836489 2026-02-13T02:05:45.644 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836489 2026-02-13T02:05:45.644 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489' 2026-02-13T02:05:45.644 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:05:45.645 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:05:45.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705672 2026-02-13T02:05:45.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705672 2026-02-13T02:05:45.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-38654705672' 2026-02-13T02:05:45.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:05:45.720 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:05:45.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574855 2026-02-13T02:05:45.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574855 2026-02-13T02:05:45.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-38654705672 2-55834574855' 2026-02-13T02:05:45.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:05:45.793 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:05:45.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411334 2026-02-13T02:05:45.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411334 2026-02-13T02:05:45.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-38654705672 2-55834574855 3-77309411334' 2026-02-13T02:05:45.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:05:45.865 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-13T02:05:45.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280518 2026-02-13T02:05:45.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280518 2026-02-13T02:05:45.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-38654705672 2-55834574855 3-77309411334 4-94489280518' 2026-02-13T02:05:45.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:05:45.937 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-13T02:05:46.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964116997 2026-02-13T02:05:46.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964116997 2026-02-13T02:05:46.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-38654705672 2-55834574855 3-77309411334 4-94489280518 5-115964116997' 2026-02-13T02:05:46.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:05:46.010 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-13T02:05:46.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953476 2026-02-13T02:05:46.082 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953476 2026-02-13T02:05:46.082 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-38654705672 2-55834574855 3-77309411334 4-94489280518 5-115964116997 6-137438953476' 2026-02-13T02:05:46.082 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:05:46.082 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-13T02:05:46.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789956 2026-02-13T02:05:46.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789956 2026-02-13T02:05:46.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-38654705672 2-55834574855 3-77309411334 4-94489280518 5-115964116997 6-137438953476 7-158913789956' 2026-02-13T02:05:46.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:05:46.153 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-13T02:05:46.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626435 2026-02-13T02:05:46.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626435 2026-02-13T02:05:46.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-38654705672 2-55834574855 3-77309411334 4-94489280518 5-115964116997 6-137438953476 7-158913789956 8-180388626435' 2026-02-13T02:05:46.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:05:46.226 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-13T02:05:46.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=201863462914 2026-02-13T02:05:46.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 201863462914 2026-02-13T02:05:46.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-38654705672 2-55834574855 3-77309411334 4-94489280518 5-115964116997 6-137438953476 7-158913789956 8-180388626435 9-201863462914' 2026-02-13T02:05:46.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:05:46.298 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836489 2026-02-13T02:05:46.298 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:05:46.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:05:46.299 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836489 2026-02-13T02:05:46.299 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:05:46.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836489 2026-02-13T02:05:46.301 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836489 2026-02-13T02:05:46.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2026-02-13T02:05:46.302 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:05:46.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836489 2026-02-13T02:05:46.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:05:47.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:05:47.477 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:05:47.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836489 2026-02-13T02:05:47.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:05:47.637 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705672 2026-02-13T02:05:47.637 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:05:47.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:05:47.639 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705672 2026-02-13T02:05:47.639 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:05:47.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705672 2026-02-13T02:05:47.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705672' 2026-02-13T02:05:47.641 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705672 2026-02-13T02:05:47.641 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:05:47.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705672 -lt 38654705672 2026-02-13T02:05:47.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:05:47.808 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574855 2026-02-13T02:05:47.808 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:05:47.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:05:47.810 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574855 2026-02-13T02:05:47.810 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:05:47.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574855 2026-02-13T02:05:47.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574855' 2026-02-13T02:05:47.812 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574855 2026-02-13T02:05:47.813 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:05:47.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574855 -lt 55834574855 2026-02-13T02:05:47.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:05:47.983 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-77309411334 2026-02-13T02:05:47.983 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:05:47.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:05:47.985 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-77309411334 2026-02-13T02:05:47.985 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:05:47.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=77309411334 2026-02-13T02:05:47.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 77309411334' 2026-02-13T02:05:47.987 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.3 seq 77309411334 2026-02-13T02:05:47.987 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:05:48.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 77309411335 -lt 77309411334 2026-02-13T02:05:48.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:05:48.160 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-94489280518 2026-02-13T02:05:48.160 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:05:48.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-13T02:05:48.162 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-94489280518 2026-02-13T02:05:48.162 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:05:48.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280518 2026-02-13T02:05:48.164 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.4 seq 94489280518 2026-02-13T02:05:48.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 94489280518' 2026-02-13T02:05:48.165 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-13T02:05:48.332 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280518 -lt 94489280518 2026-02-13T02:05:48.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:05:48.333 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-115964116997 2026-02-13T02:05:48.333 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:05:48.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-13T02:05:48.335 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-115964116997 2026-02-13T02:05:48.335 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:05:48.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=115964116997 2026-02-13T02:05:48.337 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.5 seq 115964116997 2026-02-13T02:05:48.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 115964116997' 2026-02-13T02:05:48.337 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-13T02:05:48.512 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 115964116997 -lt 115964116997 2026-02-13T02:05:48.512 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:05:48.513 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953476 2026-02-13T02:05:48.513 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:05:48.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-13T02:05:48.515 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953476 2026-02-13T02:05:48.515 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:05:48.516 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953476 2026-02-13T02:05:48.516 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953476' 2026-02-13T02:05:48.517 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.6 seq 137438953476 2026-02-13T02:05:48.517 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-13T02:05:48.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953477 -lt 137438953476 2026-02-13T02:05:48.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:05:48.690 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789956 2026-02-13T02:05:48.690 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:05:48.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-13T02:05:48.692 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789956 2026-02-13T02:05:48.692 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:05:48.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789956 2026-02-13T02:05:48.694 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789956' 2026-02-13T02:05:48.694 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.7 seq 158913789956 2026-02-13T02:05:48.694 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-13T02:05:48.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789956 -lt 158913789956 2026-02-13T02:05:48.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:05:48.863 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626435 2026-02-13T02:05:48.863 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:05:48.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-13T02:05:48.865 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626435 2026-02-13T02:05:48.866 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:05:48.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626435 2026-02-13T02:05:48.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626435' 2026-02-13T02:05:48.867 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.8 seq 180388626435 2026-02-13T02:05:48.867 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-13T02:05:49.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626435 -lt 180388626435 2026-02-13T02:05:49.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:05:49.035 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-201863462914 2026-02-13T02:05:49.035 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:05:49.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-13T02:05:49.037 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-201863462914 2026-02-13T02:05:49.037 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:05:49.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=201863462914 2026-02-13T02:05:49.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 201863462914' 2026-02-13T02:05:49.039 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.9 seq 201863462914 2026-02-13T02:05:49.039 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-13T02:05:49.203 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 201863462914 -lt 201863462914 2026-02-13T02:05:49.204 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:05:49.205 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:05:49.205 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:05:49.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-13T02:05:49.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:05:49.400 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:05:49.400 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:05:49.400 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:05:49.401 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:05:49.401 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:05:49.401 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:05:49.556 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-13T02:05:49.557 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:05:49.557 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:05:49.557 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:05:49.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-13T02:05:49.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:05:49.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:05:49.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:896: corrupt_and_repair_lrc: create_ec_pool ecpool false k=4 m=2 l=3 plugin=lrc 2026-02-13T02:05:49.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2522: create_ec_pool: local pool_name=ecpool 2026-02-13T02:05:49.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2523: create_ec_pool: shift 2026-02-13T02:05:49.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2524: create_ec_pool: local allow_overwrites=false 2026-02-13T02:05:49.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2525: create_ec_pool: shift 2026-02-13T02:05:49.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2527: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2026-02-13T02:05:49.965 INFO:tasks.workunit.client.0.trial112.stderr:load: isa 2026-02-13T02:05:49.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2529: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2026-02-13T02:05:49.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2026-02-13T02:05:50.270 INFO:tasks.workunit.client.0.trial112.stderr:pool 'ecpool' created 2026-02-13T02:05:50.280 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:05:51.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2531: create_ec_pool: '[' false = true ']' 2026-02-13T02:05:51.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2535: create_ec_pool: wait_for_clean 2026-02-13T02:05:51.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:05:51.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:05:51.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:05:51.282 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:05:51.283 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:05:51.283 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:05:51.283 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:05:51.284 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:05:51.284 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:05:51.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:05:51.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:05:51.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:05:51.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:05:51.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:05:51.392 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:05:51.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:05:51.554 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:05:51.555 INFO:tasks.workunit.client.0.trial112.stderr:2 2026-02-13T02:05:51.555 INFO:tasks.workunit.client.0.trial112.stderr:3 2026-02-13T02:05:51.555 INFO:tasks.workunit.client.0.trial112.stderr:4 2026-02-13T02:05:51.555 INFO:tasks.workunit.client.0.trial112.stderr:5 2026-02-13T02:05:51.555 INFO:tasks.workunit.client.0.trial112.stderr:6 2026-02-13T02:05:51.555 INFO:tasks.workunit.client.0.trial112.stderr:7 2026-02-13T02:05:51.555 INFO:tasks.workunit.client.0.trial112.stderr:8 2026-02-13T02:05:51.555 INFO:tasks.workunit.client.0.trial112.stderr:9' 2026-02-13T02:05:51.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:05:51.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:05:51.556 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:05:51.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836491 2026-02-13T02:05:51.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836491 2026-02-13T02:05:51.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491' 2026-02-13T02:05:51.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:05:51.628 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:05:51.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705674 2026-02-13T02:05:51.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705674 2026-02-13T02:05:51.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705674' 2026-02-13T02:05:51.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:05:51.703 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:05:51.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574857 2026-02-13T02:05:51.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574857 2026-02-13T02:05:51.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705674 2-55834574857' 2026-02-13T02:05:51.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:05:51.780 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:05:51.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411337 2026-02-13T02:05:51.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411337 2026-02-13T02:05:51.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705674 2-55834574857 3-77309411337' 2026-02-13T02:05:51.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:05:51.853 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-13T02:05:51.925 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280520 2026-02-13T02:05:51.925 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280520 2026-02-13T02:05:51.925 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705674 2-55834574857 3-77309411337 4-94489280520' 2026-02-13T02:05:51.926 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:05:51.926 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-13T02:05:51.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964116999 2026-02-13T02:05:51.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964116999 2026-02-13T02:05:51.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705674 2-55834574857 3-77309411337 4-94489280520 5-115964116999' 2026-02-13T02:05:51.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:05:51.998 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-13T02:05:52.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953478 2026-02-13T02:05:52.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953478 2026-02-13T02:05:52.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705674 2-55834574857 3-77309411337 4-94489280520 5-115964116999 6-137438953478' 2026-02-13T02:05:52.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:05:52.069 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-13T02:05:52.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789958 2026-02-13T02:05:52.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789958 2026-02-13T02:05:52.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705674 2-55834574857 3-77309411337 4-94489280520 5-115964116999 6-137438953478 7-158913789958' 2026-02-13T02:05:52.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:05:52.142 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-13T02:05:52.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626437 2026-02-13T02:05:52.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626437 2026-02-13T02:05:52.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705674 2-55834574857 3-77309411337 4-94489280520 5-115964116999 6-137438953478 7-158913789958 8-180388626437' 2026-02-13T02:05:52.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:05:52.217 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-13T02:05:52.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=201863462916 2026-02-13T02:05:52.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 201863462916 2026-02-13T02:05:52.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705674 2-55834574857 3-77309411337 4-94489280520 5-115964116999 6-137438953478 7-158913789958 8-180388626437 9-201863462916' 2026-02-13T02:05:52.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:05:52.288 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836491 2026-02-13T02:05:52.288 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:05:52.290 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:05:52.291 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836491 2026-02-13T02:05:52.291 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:05:52.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836491 2026-02-13T02:05:52.292 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836491 2026-02-13T02:05:52.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2026-02-13T02:05:52.293 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:05:52.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836490 -lt 21474836491 2026-02-13T02:05:52.469 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:05:53.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:05:53.470 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:05:53.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836491 2026-02-13T02:05:53.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:05:53.629 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705674 2026-02-13T02:05:53.629 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:05:53.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:05:53.631 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705674 2026-02-13T02:05:53.631 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:05:53.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705674 2026-02-13T02:05:53.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705674' 2026-02-13T02:05:53.634 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705674 2026-02-13T02:05:53.634 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:05:53.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705674 -lt 38654705674 2026-02-13T02:05:53.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:05:53.797 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574857 2026-02-13T02:05:53.797 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:05:53.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:05:53.798 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574857 2026-02-13T02:05:53.799 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:05:53.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574857 2026-02-13T02:05:53.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574857' 2026-02-13T02:05:53.801 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574857 2026-02-13T02:05:53.801 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:05:53.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574858 -lt 55834574857 2026-02-13T02:05:53.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:05:53.961 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-77309411337 2026-02-13T02:05:53.961 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:05:53.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:05:53.963 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-77309411337 2026-02-13T02:05:53.963 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:05:53.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=77309411337 2026-02-13T02:05:53.965 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.3 seq 77309411337 2026-02-13T02:05:53.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 77309411337' 2026-02-13T02:05:53.965 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:05:54.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 77309411337 -lt 77309411337 2026-02-13T02:05:54.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:05:54.130 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-94489280520 2026-02-13T02:05:54.131 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:05:54.132 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-13T02:05:54.132 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-94489280520 2026-02-13T02:05:54.133 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:05:54.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280520 2026-02-13T02:05:54.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 94489280520' 2026-02-13T02:05:54.135 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.4 seq 94489280520 2026-02-13T02:05:54.135 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-13T02:05:54.295 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280520 -lt 94489280520 2026-02-13T02:05:54.295 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:05:54.295 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-115964116999 2026-02-13T02:05:54.296 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:05:54.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-13T02:05:54.297 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-115964116999 2026-02-13T02:05:54.298 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:05:54.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=115964116999 2026-02-13T02:05:54.299 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.5 seq 115964116999 2026-02-13T02:05:54.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 115964116999' 2026-02-13T02:05:54.300 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-13T02:05:54.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 115964116999 -lt 115964116999 2026-02-13T02:05:54.463 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:05:54.463 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953478 2026-02-13T02:05:54.463 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:05:54.465 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-13T02:05:54.465 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953478 2026-02-13T02:05:54.466 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:05:54.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953478 2026-02-13T02:05:54.467 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.6 seq 137438953478 2026-02-13T02:05:54.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953478' 2026-02-13T02:05:54.468 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-13T02:05:54.645 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953479 -lt 137438953478 2026-02-13T02:05:54.645 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:05:54.646 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789958 2026-02-13T02:05:54.646 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:05:54.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-13T02:05:54.648 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789958 2026-02-13T02:05:54.648 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:05:54.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789958 2026-02-13T02:05:54.650 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.7 seq 158913789958 2026-02-13T02:05:54.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789958' 2026-02-13T02:05:54.650 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-13T02:05:54.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789958 -lt 158913789958 2026-02-13T02:05:54.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:05:54.810 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626437 2026-02-13T02:05:54.810 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:05:54.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-13T02:05:54.812 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626437 2026-02-13T02:05:54.812 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:05:54.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626437 2026-02-13T02:05:54.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626437' 2026-02-13T02:05:54.814 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.8 seq 180388626437 2026-02-13T02:05:54.815 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-13T02:05:54.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626437 -lt 180388626437 2026-02-13T02:05:54.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:05:54.978 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-201863462916 2026-02-13T02:05:54.978 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:05:54.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-13T02:05:54.980 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-201863462916 2026-02-13T02:05:54.980 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:05:54.981 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=201863462916 2026-02-13T02:05:54.982 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.9 seq 201863462916 2026-02-13T02:05:54.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 201863462916' 2026-02-13T02:05:54.982 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-13T02:05:55.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 201863462916 -lt 201863462916 2026-02-13T02:05:55.147 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:05:55.148 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:05:55.148 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:05:55.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:05:55.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:05:55.353 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:05:55.353 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:05:55.353 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:05:55.353 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:05:55.354 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:05:55.354 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:05:55.513 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:05:55.514 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:05:55.514 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:05:55.515 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:05:55.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:05:55.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:05:55.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:05:55.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2536: create_ec_pool: return 0 2026-02-13T02:05:55.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:897: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2026-02-13T02:05:55.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:308: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2026-02-13T02:05:55.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:309: corrupt_and_repair_erasure_coded: local poolname=ecpool 2026-02-13T02:05:55.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:311: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2026-02-13T02:05:55.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:05:55.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-13T02:05:55.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2026-02-13T02:05:55.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:05:55.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:05:55.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:05:55.918 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:05:55.928 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:05:56.122 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:05:56.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:05:56.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:05:56.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-13T02:05:56.160 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:313: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2026-02-13T02:05:56.160 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2026-02-13T02:05:56.160 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-13T02:05:56.160 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T02:05:56.160 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2026-02-13T02:05:56.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:313: corrupt_and_repair_erasure_coded: local primary=3 2026-02-13T02:05:56.328 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2026-02-13T02:05:56.328 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: sed -e s/3// 2026-02-13T02:05:56.328 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2026-02-13T02:05:56.329 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-13T02:05:56.329 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2026-02-13T02:05:56.329 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-13T02:05:56.493 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-13T02:05:56.493 INFO:tasks.workunit.client.0.trial112.stderr:5 2026-02-13T02:05:56.493 INFO:tasks.workunit.client.0.trial112.stderr:9 2026-02-13T02:05:56.494 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:05:56.494 INFO:tasks.workunit.client.0.trial112.stderr:6 2026-02-13T02:05:56.494 INFO:tasks.workunit.client.0.trial112.stderr:2 2026-02-13T02:05:56.494 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:05:56.494 INFO:tasks.workunit.client.0.trial112.stderr:7' 2026-02-13T02:05:56.494 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 9 0 6 2 1 7 2026-02-13T02:05:56.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: osds=('5' '9' '0' '6' '2' '1' '7') 2026-02-13T02:05:56.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: local -a osds 2026-02-13T02:05:56.497 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:315: corrupt_and_repair_erasure_coded: local not_primary_first=5 2026-02-13T02:05:56.497 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:316: corrupt_and_repair_erasure_coded: local not_primary_second=9 2026-02-13T02:05:56.497 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:319: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2026-02-13T02:05:56.497 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:286: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2026-02-13T02:05:56.497 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: corrupt_and_repair_one: local poolname=ecpool 2026-02-13T02:05:56.497 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:288: corrupt_and_repair_one: local osd=3 2026-02-13T02:05:56.497 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:293: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-13T02:05:56.497 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:05:56.497 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:05:56.497 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-13T02:05:56.497 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:05:56.498 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2026-02-13T02:05:56.498 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:05:56.498 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:05:56.498 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-13T02:05:56.498 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:05:56.498 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-13T02:05:56.498 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:05:56.498 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:05:56.498 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:05:56.498 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:05:56.498 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:05:56.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:05:56.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2026-02-13T02:05:56.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:05:56.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:05:56.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-13T02:05:56.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:05:56.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:05:56.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2026-02-13T02:05:56.996 INFO:tasks.workunit.client.0.trial112.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2026-02-13T02:05:57.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-13T02:05:57.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:05:57.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:05:57.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-13T02:05:57.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:05:57.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:05:57.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:05:57.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:05:57.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:05:57.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:05:57.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-13T02:05:57.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-13T02:05:57.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:05:57.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:05:57.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:05:57.093 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:05:57.094 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:05:57.094 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:05:57.094 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:05:57.094 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:05:57.094 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:05:57.094 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:05:57.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:05:57.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:05:57.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:05:57.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:05:57.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:05:57.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:05:57.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:05:57.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:05:57.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:05:57.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:05:57.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-13T02:05:57.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-13T02:05:57.096 INFO:tasks.workunit.client.0.trial112.stderr:start osd.3 2026-02-13T02:05:57.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:05:57.096 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-13T02:05:57.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-13T02:05:57.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:05:57.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:05:57.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:05:57.112 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:57.100+0000 7efc387afa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:57.114 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:57.104+0000 7efc387afa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:57.115 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:57.104+0000 7efc387afa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:05:57.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-13T02:05:57.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:05:57.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-13T02:05:57.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:05:57.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:05:57.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:05:57.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:05:57.259 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:05:57.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:05:57.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:05:57.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:05:57.559 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:57.548+0000 7efc387afa40 -1 Falling back to public interface 2026-02-13T02:05:58.180 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:05:58.168+0000 7efc387afa40 -1 osd.3 62 log_to_monitors true 2026-02-13T02:05:58.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:05:58.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:05:58.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:05:58.427 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:05:58.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:05:58.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:05:58.587 INFO:tasks.workunit.client.0.trial112.stderr:osd.3 up in weight 1 up_from 66 up_thru 66 down_at 63 last_clean_interval [18,62) [v2:127.0.0.1:6826/101178732,v1:127.0.0.1:6827/101178732] [v2:127.0.0.1:6828/101178732,v1:127.0.0.1:6829/101178732] exists,up ad30d011-cd62-4310-8456-c2d78faa8a93 2026-02-13T02:05:58.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:05:58.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:05:58.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:05:58.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:05:58.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:05:58.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:05:58.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:05:58.587 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:05:58.588 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:05:58.588 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:05:58.588 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:05:58.588 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:05:58.588 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:05:58.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:05:58.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:05:58.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:05:58.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:05:58.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:05:58.698 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:05:58.859 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:05:58.860 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:05:58.860 INFO:tasks.workunit.client.0.trial112.stderr:2 2026-02-13T02:05:58.860 INFO:tasks.workunit.client.0.trial112.stderr:3 2026-02-13T02:05:58.860 INFO:tasks.workunit.client.0.trial112.stderr:4 2026-02-13T02:05:58.860 INFO:tasks.workunit.client.0.trial112.stderr:5 2026-02-13T02:05:58.860 INFO:tasks.workunit.client.0.trial112.stderr:6 2026-02-13T02:05:58.860 INFO:tasks.workunit.client.0.trial112.stderr:7 2026-02-13T02:05:58.860 INFO:tasks.workunit.client.0.trial112.stderr:8 2026-02-13T02:05:58.860 INFO:tasks.workunit.client.0.trial112.stderr:9' 2026-02-13T02:05:58.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:05:58.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:05:58.860 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:05:58.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836493 2026-02-13T02:05:58.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836493 2026-02-13T02:05:58.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493' 2026-02-13T02:05:58.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:05:58.935 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:05:59.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705677 2026-02-13T02:05:59.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705677 2026-02-13T02:05:59.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-38654705677' 2026-02-13T02:05:59.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:05:59.008 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:05:59.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574860 2026-02-13T02:05:59.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574860 2026-02-13T02:05:59.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-38654705677 2-55834574860' 2026-02-13T02:05:59.086 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:05:59.086 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:05:59.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=283467841538 2026-02-13T02:05:59.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 283467841538 2026-02-13T02:05:59.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-38654705677 2-55834574860 3-283467841538' 2026-02-13T02:05:59.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:05:59.160 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-13T02:05:59.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280522 2026-02-13T02:05:59.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280522 2026-02-13T02:05:59.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-38654705677 2-55834574860 3-283467841538 4-94489280522' 2026-02-13T02:05:59.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:05:59.234 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-13T02:05:59.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964117002 2026-02-13T02:05:59.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964117002 2026-02-13T02:05:59.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-38654705677 2-55834574860 3-283467841538 4-94489280522 5-115964117002' 2026-02-13T02:05:59.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:05:59.308 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-13T02:05:59.381 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953481 2026-02-13T02:05:59.381 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953481 2026-02-13T02:05:59.381 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-38654705677 2-55834574860 3-283467841538 4-94489280522 5-115964117002 6-137438953481' 2026-02-13T02:05:59.381 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:05:59.382 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-13T02:05:59.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789960 2026-02-13T02:05:59.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789960 2026-02-13T02:05:59.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-38654705677 2-55834574860 3-283467841538 4-94489280522 5-115964117002 6-137438953481 7-158913789960' 2026-02-13T02:05:59.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:05:59.454 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-13T02:05:59.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626439 2026-02-13T02:05:59.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626439 2026-02-13T02:05:59.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-38654705677 2-55834574860 3-283467841538 4-94489280522 5-115964117002 6-137438953481 7-158913789960 8-180388626439' 2026-02-13T02:05:59.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:05:59.526 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-13T02:05:59.599 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=201863462919 2026-02-13T02:05:59.599 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 201863462919 2026-02-13T02:05:59.599 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-38654705677 2-55834574860 3-283467841538 4-94489280522 5-115964117002 6-137438953481 7-158913789960 8-180388626439 9-201863462919' 2026-02-13T02:05:59.599 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:05:59.600 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836493 2026-02-13T02:05:59.600 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:05:59.601 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:05:59.602 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836493 2026-02-13T02:05:59.602 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:05:59.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836493 2026-02-13T02:05:59.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2026-02-13T02:05:59.603 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 21474836493 2026-02-13T02:05:59.604 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:05:59.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836493 -lt 21474836493 2026-02-13T02:05:59.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:05:59.773 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705677 2026-02-13T02:05:59.773 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:05:59.774 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:05:59.775 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705677 2026-02-13T02:05:59.775 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:05:59.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705677 2026-02-13T02:05:59.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705677' 2026-02-13T02:05:59.777 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 38654705677 2026-02-13T02:05:59.777 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:05:59.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705677 -lt 38654705677 2026-02-13T02:05:59.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:05:59.952 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574860 2026-02-13T02:05:59.952 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:05:59.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:05:59.954 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574860 2026-02-13T02:05:59.954 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:05:59.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574860 2026-02-13T02:05:59.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574860' 2026-02-13T02:05:59.956 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 55834574860 2026-02-13T02:05:59.956 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:06:00.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574860 -lt 55834574860 2026-02-13T02:06:00.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:00.121 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-283467841538 2026-02-13T02:06:00.121 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:00.122 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:06:00.123 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-283467841538 2026-02-13T02:06:00.123 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:00.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=283467841538 2026-02-13T02:06:00.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 283467841538' 2026-02-13T02:06:00.125 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.3 seq 283467841538 2026-02-13T02:06:00.125 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:06:00.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 283467841538 -lt 283467841538 2026-02-13T02:06:00.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:00.298 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-94489280522 2026-02-13T02:06:00.298 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:00.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-13T02:06:00.300 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-94489280522 2026-02-13T02:06:00.300 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:00.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280522 2026-02-13T02:06:00.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 94489280522' 2026-02-13T02:06:00.302 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.4 seq 94489280522 2026-02-13T02:06:00.302 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-13T02:06:00.463 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280521 -lt 94489280522 2026-02-13T02:06:00.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:06:01.465 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:06:01.465 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-13T02:06:01.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280523 -lt 94489280522 2026-02-13T02:06:01.624 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:01.624 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-115964117002 2026-02-13T02:06:01.624 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:01.625 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-13T02:06:01.626 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-115964117002 2026-02-13T02:06:01.627 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:01.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=115964117002 2026-02-13T02:06:01.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 115964117002' 2026-02-13T02:06:01.628 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.5 seq 115964117002 2026-02-13T02:06:01.629 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-13T02:06:01.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 115964117002 -lt 115964117002 2026-02-13T02:06:01.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:01.794 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953481 2026-02-13T02:06:01.794 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:01.795 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-13T02:06:01.796 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953481 2026-02-13T02:06:01.796 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:01.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953481 2026-02-13T02:06:01.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953481' 2026-02-13T02:06:01.798 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.6 seq 137438953481 2026-02-13T02:06:01.799 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-13T02:06:01.959 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953481 -lt 137438953481 2026-02-13T02:06:01.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:01.960 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789960 2026-02-13T02:06:01.960 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:01.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-13T02:06:01.962 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789960 2026-02-13T02:06:01.962 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:01.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789960 2026-02-13T02:06:01.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789960' 2026-02-13T02:06:01.964 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.7 seq 158913789960 2026-02-13T02:06:01.965 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-13T02:06:02.138 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789961 -lt 158913789960 2026-02-13T02:06:02.138 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:02.139 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626439 2026-02-13T02:06:02.139 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:02.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-13T02:06:02.141 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626439 2026-02-13T02:06:02.141 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:02.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626439 2026-02-13T02:06:02.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626439' 2026-02-13T02:06:02.143 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.8 seq 180388626439 2026-02-13T02:06:02.143 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-13T02:06:02.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626440 -lt 180388626439 2026-02-13T02:06:02.315 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:02.315 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-201863462919 2026-02-13T02:06:02.315 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:02.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-13T02:06:02.317 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-201863462919 2026-02-13T02:06:02.318 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:02.319 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=201863462919 2026-02-13T02:06:02.319 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 201863462919' 2026-02-13T02:06:02.319 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.9 seq 201863462919 2026-02-13T02:06:02.320 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-13T02:06:02.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 201863462919 -lt 201863462919 2026-02-13T02:06:02.481 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:06:02.482 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:06:02.482 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:06:02.680 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:06:02.681 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:06:02.681 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:06:02.681 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:06:02.681 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:06:02.681 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:06:02.681 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:06:02.681 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:06:02.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:06:02.843 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:06:02.843 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:06:02.843 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:06:03.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:06:03.042 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:06:03.042 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:06:03.042 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: get_pg ecpool SOMETHING 2026-02-13T02:06:03.042 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-13T02:06:03.042 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-13T02:06:03.043 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-13T02:06:03.043 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T02:06:03.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: local pg=2.0 2026-02-13T02:06:03.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: corrupt_and_repair_one: repair 2.0 2026-02-13T02:06:03.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-13T02:06:03.201 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-13T02:06:03.201 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:06:03.201 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:06:03.201 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:06:03.201 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:06:03.359 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-13T02:05:50.262490+0000 2026-02-13T02:06:03.359 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-13T02:06:03.513 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-13T02:06:03.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-13T02:05:50.262490+0000 2026-02-13T02:06:03.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-13T02:06:03.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:05:50.262490+0000 2026-02-13T02:06:03.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:06:03.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:06:03.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:06:03.524 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:06:03.524 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:06:03.524 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:06:03.525 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:06:03.525 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:06:03.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:05:50.262490+0000 '>' 2026-02-13T02:05:50.262490+0000 2026-02-13T02:06:03.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:06:04.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:06:04.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:06:04.684 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:06:04.685 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:06:04.685 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:06:04.685 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:06:04.685 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:06:04.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:05:50.262490+0000 '>' 2026-02-13T02:05:50.262490+0000 2026-02-13T02:06:04.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:06:05.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:06:05.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:06:05.842 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:06:05.842 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:06:05.842 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:06:05.843 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:06:05.843 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:06:06.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:05:50.262490+0000 '>' 2026-02-13T02:05:50.262490+0000 2026-02-13T02:06:06.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:06:07.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:06:07.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:06:07.008 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:06:07.008 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:06:07.008 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:06:07.009 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:06:07.009 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:06:07.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:05:50.262490+0000 '>' 2026-02-13T02:05:50.262490+0000 2026-02-13T02:06:07.170 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:06:08.170 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:06:08.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:06:08.171 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:06:08.171 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:06:08.171 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:06:08.172 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:06:08.172 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:06:08.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:05:50.262490+0000 '>' 2026-02-13T02:05:50.262490+0000 2026-02-13T02:06:08.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:06:09.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:06:09.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:06:09.336 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:06:09.336 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:06:09.336 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:06:09.336 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:06:09.336 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:06:09.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:06:04.033487+0000 '>' 2026-02-13T02:05:50.262490+0000 2026-02-13T02:06:09.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:06:09.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-13T02:06:09.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:06:09.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:06:09.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2026-02-13T02:06:09.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:06:09.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-13T02:06:09.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:06:09.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:06:09.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2026-02-13T02:06:09.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:06:09.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2026-02-13T02:06:09.502 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:06:09.502 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:06:09.502 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:06:09.502 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:06:09.502 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:06:09.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:06:09.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-13T02:06:09.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:06:09.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:06:09.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2026-02-13T02:06:09.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:06:09.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2026-02-13T02:06:09.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2026-02-13T02:06:09.880 INFO:tasks.workunit.client.0.trial112.stdout:_ 2026-02-13T02:06:09.881 INFO:tasks.workunit.client.0.trial112.stdout:hinfo_key 2026-02-13T02:06:09.881 INFO:tasks.workunit.client.0.trial112.stdout:snapset 2026-02-13T02:06:09.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2026-02-13T02:06:09.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:06:09.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:06:09.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2026-02-13T02:06:09.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:06:09.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2026-02-13T02:06:09.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:06:09.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:06:09.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:06:09.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:06:09.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2026-02-13T02:06:09.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2026-02-13T02:06:09.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:06:09.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:06:09.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:06:09.991 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:06:09.991 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:06:09.992 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:06:09.992 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:06:09.992 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:06:09.992 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:06:09.992 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:06:09.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:06:09.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:06:09.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:06:09.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:06:09.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:06:09.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:06:09.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:06:09.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:06:09.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:06:09.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:06:09.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2026-02-13T02:06:09.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2026-02-13T02:06:09.993 INFO:tasks.workunit.client.0.trial112.stderr:start osd.9 2026-02-13T02:06:09.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:06:09.993 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2026-02-13T02:06:09.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2026-02-13T02:06:09.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:06:09.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:06:09.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:06:10.007 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:06:09.996+0000 7f457a469a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:06:10.008 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:06:09.996+0000 7f457a469a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:06:10.009 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:06:10.000+0000 7f457a469a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:06:10.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2026-02-13T02:06:10.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:06:10.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2026-02-13T02:06:10.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:06:10.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:06:10.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:06:10.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:06:10.150 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:06:10.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:06:10.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-13T02:06:10.315 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:06:10.399 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:06:10.388+0000 7f457a469a40 -1 Falling back to public interface 2026-02-13T02:06:11.071 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:06:11.060+0000 7f457a469a40 -1 osd.9 69 log_to_monitors true 2026-02-13T02:06:11.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:06:11.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:06:11.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:06:11.317 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:06:11.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:06:11.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-13T02:06:11.475 INFO:tasks.workunit.client.0.trial112.stderr:osd.9 up in weight 1 up_from 72 up_thru 0 down_at 69 last_clean_interval [47,69) [v2:127.0.0.1:6874/3110900292,v1:127.0.0.1:6875/3110900292] [v2:127.0.0.1:6876/3110900292,v1:127.0.0.1:6877/3110900292] exists,up 31da8501-3453-451f-af71-20390baae5a0 2026-02-13T02:06:11.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:06:11.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:06:11.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:06:11.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:06:11.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:06:11.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:06:11.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:06:11.477 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:06:11.477 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:06:11.477 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:06:11.477 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:06:11.477 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:06:11.477 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:06:11.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:06:11.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:06:11.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:06:11.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:06:11.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:06:11.589 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:06:11.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:06:11.752 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:06:11.752 INFO:tasks.workunit.client.0.trial112.stderr:2 2026-02-13T02:06:11.752 INFO:tasks.workunit.client.0.trial112.stderr:3 2026-02-13T02:06:11.753 INFO:tasks.workunit.client.0.trial112.stderr:4 2026-02-13T02:06:11.753 INFO:tasks.workunit.client.0.trial112.stderr:5 2026-02-13T02:06:11.753 INFO:tasks.workunit.client.0.trial112.stderr:6 2026-02-13T02:06:11.753 INFO:tasks.workunit.client.0.trial112.stderr:7 2026-02-13T02:06:11.753 INFO:tasks.workunit.client.0.trial112.stderr:8 2026-02-13T02:06:11.753 INFO:tasks.workunit.client.0.trial112.stderr:9' 2026-02-13T02:06:11.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:06:11.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:11.754 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:06:11.827 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836497 2026-02-13T02:06:11.827 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836497 2026-02-13T02:06:11.827 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497' 2026-02-13T02:06:11.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:11.828 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:06:11.899 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705680 2026-02-13T02:06:11.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705680 2026-02-13T02:06:11.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-38654705680' 2026-02-13T02:06:11.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:11.900 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:06:11.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574863 2026-02-13T02:06:11.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574863 2026-02-13T02:06:11.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-38654705680 2-55834574863' 2026-02-13T02:06:11.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:11.974 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:06:12.049 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=283467841541 2026-02-13T02:06:12.049 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 283467841541 2026-02-13T02:06:12.049 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-38654705680 2-55834574863 3-283467841541' 2026-02-13T02:06:12.049 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:12.049 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-13T02:06:12.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280526 2026-02-13T02:06:12.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280526 2026-02-13T02:06:12.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-38654705680 2-55834574863 3-283467841541 4-94489280526' 2026-02-13T02:06:12.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:12.120 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-13T02:06:12.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964117005 2026-02-13T02:06:12.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964117005 2026-02-13T02:06:12.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-38654705680 2-55834574863 3-283467841541 4-94489280526 5-115964117005' 2026-02-13T02:06:12.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:12.194 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-13T02:06:12.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953485 2026-02-13T02:06:12.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953485 2026-02-13T02:06:12.266 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-38654705680 2-55834574863 3-283467841541 4-94489280526 5-115964117005 6-137438953485' 2026-02-13T02:06:12.266 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:12.266 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-13T02:06:12.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789964 2026-02-13T02:06:12.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789964 2026-02-13T02:06:12.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-38654705680 2-55834574863 3-283467841541 4-94489280526 5-115964117005 6-137438953485 7-158913789964' 2026-02-13T02:06:12.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:12.338 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-13T02:06:12.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626443 2026-02-13T02:06:12.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626443 2026-02-13T02:06:12.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-38654705680 2-55834574863 3-283467841541 4-94489280526 5-115964117005 6-137438953485 7-158913789964 8-180388626443' 2026-02-13T02:06:12.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:12.413 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-13T02:06:12.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=309237645314 2026-02-13T02:06:12.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 309237645314 2026-02-13T02:06:12.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-38654705680 2-55834574863 3-283467841541 4-94489280526 5-115964117005 6-137438953485 7-158913789964 8-180388626443 9-309237645314' 2026-02-13T02:06:12.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:12.485 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836497 2026-02-13T02:06:12.485 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:12.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:06:12.487 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836497 2026-02-13T02:06:12.487 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:12.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836497 2026-02-13T02:06:12.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2026-02-13T02:06:12.489 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 21474836497 2026-02-13T02:06:12.489 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:06:12.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836496 -lt 21474836497 2026-02-13T02:06:12.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:06:13.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:06:13.653 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:06:13.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836497 -lt 21474836497 2026-02-13T02:06:13.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:13.810 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705680 2026-02-13T02:06:13.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:13.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:06:13.813 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705680 2026-02-13T02:06:13.813 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:13.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705680 2026-02-13T02:06:13.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705680' 2026-02-13T02:06:13.815 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 38654705680 2026-02-13T02:06:13.815 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:06:13.980 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705680 -lt 38654705680 2026-02-13T02:06:13.980 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:13.981 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574863 2026-02-13T02:06:13.981 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:13.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:06:13.983 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574863 2026-02-13T02:06:13.983 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:13.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574863 2026-02-13T02:06:13.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574863' 2026-02-13T02:06:13.985 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 55834574863 2026-02-13T02:06:13.985 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:06:14.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574864 -lt 55834574863 2026-02-13T02:06:14.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:14.154 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-283467841541 2026-02-13T02:06:14.154 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:14.155 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:06:14.155 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-283467841541 2026-02-13T02:06:14.156 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:14.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=283467841541 2026-02-13T02:06:14.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 283467841541' 2026-02-13T02:06:14.157 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.3 seq 283467841541 2026-02-13T02:06:14.158 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:06:14.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 283467841541 -lt 283467841541 2026-02-13T02:06:14.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:14.329 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-94489280526 2026-02-13T02:06:14.329 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:14.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-13T02:06:14.331 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-94489280526 2026-02-13T02:06:14.332 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:14.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280526 2026-02-13T02:06:14.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 94489280526' 2026-02-13T02:06:14.333 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.4 seq 94489280526 2026-02-13T02:06:14.334 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-13T02:06:14.512 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280526 -lt 94489280526 2026-02-13T02:06:14.513 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:14.513 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-115964117005 2026-02-13T02:06:14.513 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:14.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-13T02:06:14.515 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-115964117005 2026-02-13T02:06:14.516 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:14.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=115964117005 2026-02-13T02:06:14.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 115964117005' 2026-02-13T02:06:14.517 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.5 seq 115964117005 2026-02-13T02:06:14.517 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-13T02:06:14.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 115964117005 -lt 115964117005 2026-02-13T02:06:14.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:14.686 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953485 2026-02-13T02:06:14.687 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:14.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-13T02:06:14.689 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953485 2026-02-13T02:06:14.689 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:14.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953485 2026-02-13T02:06:14.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953485' 2026-02-13T02:06:14.690 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.6 seq 137438953485 2026-02-13T02:06:14.691 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-13T02:06:14.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953485 -lt 137438953485 2026-02-13T02:06:14.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:14.856 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789964 2026-02-13T02:06:14.856 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:14.857 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-13T02:06:14.858 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789964 2026-02-13T02:06:14.858 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:14.859 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789964 2026-02-13T02:06:14.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789964' 2026-02-13T02:06:14.860 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.7 seq 158913789964 2026-02-13T02:06:14.860 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-13T02:06:15.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789964 -lt 158913789964 2026-02-13T02:06:15.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:15.031 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626443 2026-02-13T02:06:15.031 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:15.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-13T02:06:15.033 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626443 2026-02-13T02:06:15.033 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:15.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626443 2026-02-13T02:06:15.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626443' 2026-02-13T02:06:15.035 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.8 seq 180388626443 2026-02-13T02:06:15.035 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-13T02:06:15.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626443 -lt 180388626443 2026-02-13T02:06:15.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:15.209 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-309237645314 2026-02-13T02:06:15.209 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:15.210 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-13T02:06:15.211 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-309237645314 2026-02-13T02:06:15.211 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:15.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=309237645314 2026-02-13T02:06:15.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 309237645314' 2026-02-13T02:06:15.213 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.9 seq 309237645314 2026-02-13T02:06:15.213 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-13T02:06:15.383 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 309237645314 -lt 309237645314 2026-02-13T02:06:15.384 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:06:15.384 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:06:15.384 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:06:15.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:06:15.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:06:15.572 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:06:15.572 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:06:15.572 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:06:15.572 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:06:15.573 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:06:15.573 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:06:15.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:06:15.732 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:06:15.732 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:06:15.733 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:06:15.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:06:15.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:06:15.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:06:15.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-13T02:06:15.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-13T02:06:15.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:321: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2026-02-13T02:06:15.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:286: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2026-02-13T02:06:15.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: corrupt_and_repair_one: local poolname=ecpool 2026-02-13T02:06:15.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:288: corrupt_and_repair_one: local osd=5 2026-02-13T02:06:15.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:293: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2026-02-13T02:06:15.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:06:15.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:06:15.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2026-02-13T02:06:15.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:06:15.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2026-02-13T02:06:15.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:06:15.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:06:15.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2026-02-13T02:06:15.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:06:15.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2026-02-13T02:06:15.956 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:06:15.956 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:06:15.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:06:15.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:06:15.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:06:16.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:06:16.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2026-02-13T02:06:16.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:06:16.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:06:16.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2026-02-13T02:06:16.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:06:16.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2026-02-13T02:06:16.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2026-02-13T02:06:16.438 INFO:tasks.workunit.client.0.trial112.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2026-02-13T02:06:16.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2026-02-13T02:06:16.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:06:16.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:06:16.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2026-02-13T02:06:16.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:06:16.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2026-02-13T02:06:16.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:06:16.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:06:16.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:06:16.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:06:16.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2026-02-13T02:06:16.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2026-02-13T02:06:16.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:06:16.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:06:16.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:06:16.542 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:06:16.543 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:06:16.543 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:06:16.543 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:06:16.543 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:06:16.543 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:06:16.543 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:06:16.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:06:16.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:06:16.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:06:16.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:06:16.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:06:16.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:06:16.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:06:16.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:06:16.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:06:16.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:06:16.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2026-02-13T02:06:16.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2026-02-13T02:06:16.545 INFO:tasks.workunit.client.0.trial112.stderr:start osd.5 2026-02-13T02:06:16.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:06:16.545 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2026-02-13T02:06:16.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2026-02-13T02:06:16.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:06:16.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:06:16.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:06:16.564 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:06:16.552+0000 7fe86c0e3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:06:16.565 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:06:16.552+0000 7fe86c0e3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:06:16.566 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:06:16.556+0000 7fe86c0e3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:06:16.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2026-02-13T02:06:16.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:06:16.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-13T02:06:16.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:06:16.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:06:16.707 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:06:16.707 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:06:16.707 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:06:16.707 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:06:16.707 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-13T02:06:16.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:06:16.955 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:06:16.944+0000 7fe86c0e3a40 -1 Falling back to public interface 2026-02-13T02:06:17.521 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:06:17.508+0000 7fe86c0e3a40 -1 osd.5 74 log_to_monitors true 2026-02-13T02:06:17.869 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:06:17.870 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:06:17.870 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:06:17.870 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:06:17.870 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:06:17.870 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-13T02:06:18.030 INFO:tasks.workunit.client.0.trial112.stderr:osd.5 up in weight 1 up_from 78 up_thru 78 down_at 75 last_clean_interval [27,74) [v2:127.0.0.1:6842/2888378504,v1:127.0.0.1:6843/2888378504] [v2:127.0.0.1:6844/2888378504,v1:127.0.0.1:6845/2888378504] exists,up 670b1f5b-3687-4794-8a8a-767a3a2770e7 2026-02-13T02:06:18.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:06:18.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:06:18.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:06:18.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:06:18.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:06:18.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:06:18.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:06:18.031 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:06:18.032 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:06:18.032 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:06:18.032 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:06:18.032 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:06:18.032 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:06:18.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:06:18.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:06:18.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:06:18.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:06:18.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:06:18.140 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:06:18.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:06:18.305 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:06:18.305 INFO:tasks.workunit.client.0.trial112.stderr:2 2026-02-13T02:06:18.305 INFO:tasks.workunit.client.0.trial112.stderr:3 2026-02-13T02:06:18.305 INFO:tasks.workunit.client.0.trial112.stderr:4 2026-02-13T02:06:18.305 INFO:tasks.workunit.client.0.trial112.stderr:5 2026-02-13T02:06:18.305 INFO:tasks.workunit.client.0.trial112.stderr:6 2026-02-13T02:06:18.305 INFO:tasks.workunit.client.0.trial112.stderr:7 2026-02-13T02:06:18.305 INFO:tasks.workunit.client.0.trial112.stderr:8 2026-02-13T02:06:18.305 INFO:tasks.workunit.client.0.trial112.stderr:9' 2026-02-13T02:06:18.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:06:18.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:18.306 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:06:18.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836499 2026-02-13T02:06:18.381 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836499 2026-02-13T02:06:18.381 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499' 2026-02-13T02:06:18.381 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:18.381 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:06:18.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705682 2026-02-13T02:06:18.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705682 2026-02-13T02:06:18.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-38654705682' 2026-02-13T02:06:18.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:18.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:06:18.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574866 2026-02-13T02:06:18.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574866 2026-02-13T02:06:18.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-38654705682 2-55834574866' 2026-02-13T02:06:18.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:18.533 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:06:18.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=283467841544 2026-02-13T02:06:18.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 283467841544 2026-02-13T02:06:18.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-38654705682 2-55834574866 3-283467841544' 2026-02-13T02:06:18.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:18.605 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-13T02:06:18.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280528 2026-02-13T02:06:18.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280528 2026-02-13T02:06:18.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-38654705682 2-55834574866 3-283467841544 4-94489280528' 2026-02-13T02:06:18.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:18.677 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-13T02:06:18.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=335007449090 2026-02-13T02:06:18.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 335007449090 2026-02-13T02:06:18.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-38654705682 2-55834574866 3-283467841544 4-94489280528 5-335007449090' 2026-02-13T02:06:18.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:18.752 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-13T02:06:18.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953487 2026-02-13T02:06:18.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953487 2026-02-13T02:06:18.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-38654705682 2-55834574866 3-283467841544 4-94489280528 5-335007449090 6-137438953487' 2026-02-13T02:06:18.827 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:18.827 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-13T02:06:18.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789966 2026-02-13T02:06:18.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789966 2026-02-13T02:06:18.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-38654705682 2-55834574866 3-283467841544 4-94489280528 5-335007449090 6-137438953487 7-158913789966' 2026-02-13T02:06:18.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:18.900 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-13T02:06:18.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626445 2026-02-13T02:06:18.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626445 2026-02-13T02:06:18.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-38654705682 2-55834574866 3-283467841544 4-94489280528 5-335007449090 6-137438953487 7-158913789966 8-180388626445' 2026-02-13T02:06:18.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:18.973 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-13T02:06:19.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=309237645316 2026-02-13T02:06:19.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 309237645316 2026-02-13T02:06:19.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-38654705682 2-55834574866 3-283467841544 4-94489280528 5-335007449090 6-137438953487 7-158913789966 8-180388626445 9-309237645316' 2026-02-13T02:06:19.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:19.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836499 2026-02-13T02:06:19.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:19.048 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:06:19.048 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836499 2026-02-13T02:06:19.049 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:19.050 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836499 2026-02-13T02:06:19.050 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2026-02-13T02:06:19.050 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 21474836499 2026-02-13T02:06:19.051 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:06:19.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836498 -lt 21474836499 2026-02-13T02:06:19.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:06:20.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:06:20.209 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:06:20.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836499 -lt 21474836499 2026-02-13T02:06:20.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:20.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705682 2026-02-13T02:06:20.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:20.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:06:20.375 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705682 2026-02-13T02:06:20.376 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:20.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705682 2026-02-13T02:06:20.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705682' 2026-02-13T02:06:20.377 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 38654705682 2026-02-13T02:06:20.378 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:06:20.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705683 -lt 38654705682 2026-02-13T02:06:20.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:20.553 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574866 2026-02-13T02:06:20.553 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:20.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:06:20.555 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574866 2026-02-13T02:06:20.555 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:20.557 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574866 2026-02-13T02:06:20.557 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574866' 2026-02-13T02:06:20.557 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 55834574866 2026-02-13T02:06:20.557 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:06:20.720 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574866 -lt 55834574866 2026-02-13T02:06:20.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:20.721 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-283467841544 2026-02-13T02:06:20.721 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:20.723 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:06:20.723 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-283467841544 2026-02-13T02:06:20.724 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:20.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=283467841544 2026-02-13T02:06:20.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 283467841544' 2026-02-13T02:06:20.725 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.3 seq 283467841544 2026-02-13T02:06:20.725 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:06:20.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 283467841544 -lt 283467841544 2026-02-13T02:06:20.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:20.886 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-94489280528 2026-02-13T02:06:20.886 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:20.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-13T02:06:20.888 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-94489280528 2026-02-13T02:06:20.888 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:20.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280528 2026-02-13T02:06:20.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 94489280528' 2026-02-13T02:06:20.890 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.4 seq 94489280528 2026-02-13T02:06:20.890 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-13T02:06:21.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280528 -lt 94489280528 2026-02-13T02:06:21.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:21.063 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-335007449090 2026-02-13T02:06:21.063 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:21.064 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-13T02:06:21.065 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-335007449090 2026-02-13T02:06:21.065 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:21.066 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=335007449090 2026-02-13T02:06:21.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 335007449090' 2026-02-13T02:06:21.067 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.5 seq 335007449090 2026-02-13T02:06:21.067 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-13T02:06:21.229 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 335007449090 -lt 335007449090 2026-02-13T02:06:21.229 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:21.230 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953487 2026-02-13T02:06:21.230 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:21.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-13T02:06:21.231 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953487 2026-02-13T02:06:21.231 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:21.233 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953487 2026-02-13T02:06:21.233 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953487' 2026-02-13T02:06:21.233 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.6 seq 137438953487 2026-02-13T02:06:21.234 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-13T02:06:21.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953487 -lt 137438953487 2026-02-13T02:06:21.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:21.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789966 2026-02-13T02:06:21.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:21.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-13T02:06:21.408 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789966 2026-02-13T02:06:21.409 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:21.410 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789966 2026-02-13T02:06:21.410 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789966' 2026-02-13T02:06:21.410 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.7 seq 158913789966 2026-02-13T02:06:21.411 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-13T02:06:21.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789967 -lt 158913789966 2026-02-13T02:06:21.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:21.577 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626445 2026-02-13T02:06:21.578 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:21.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-13T02:06:21.579 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626445 2026-02-13T02:06:21.579 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:21.581 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626445 2026-02-13T02:06:21.581 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626445' 2026-02-13T02:06:21.581 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.8 seq 180388626445 2026-02-13T02:06:21.581 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-13T02:06:21.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626446 -lt 180388626445 2026-02-13T02:06:21.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:21.741 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-309237645316 2026-02-13T02:06:21.741 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:21.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-13T02:06:21.743 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-309237645316 2026-02-13T02:06:21.744 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:21.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=309237645316 2026-02-13T02:06:21.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 309237645316' 2026-02-13T02:06:21.745 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.9 seq 309237645316 2026-02-13T02:06:21.746 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-13T02:06:21.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 309237645317 -lt 309237645316 2026-02-13T02:06:21.919 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:06:21.920 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:06:21.920 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:06:22.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:06:22.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:06:22.113 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:06:22.113 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:06:22.113 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:06:22.113 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:06:22.113 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:06:22.114 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:06:22.277 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:06:22.278 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:06:22.278 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:06:22.278 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:06:22.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:06:22.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:06:22.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:06:22.480 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: get_pg ecpool SOMETHING 2026-02-13T02:06:22.480 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-13T02:06:22.480 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-13T02:06:22.481 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-13T02:06:22.481 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T02:06:22.645 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: local pg=2.0 2026-02-13T02:06:22.645 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: corrupt_and_repair_one: repair 2.0 2026-02-13T02:06:22.646 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-13T02:06:22.646 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-13T02:06:22.646 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:06:22.646 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:06:22.646 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:06:22.646 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:06:22.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-13T02:06:04.033487+0000 2026-02-13T02:06:22.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-13T02:06:22.956 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-13T02:06:22.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-13T02:06:04.033487+0000 2026-02-13T02:06:22.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-13T02:06:22.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:06:04.033487+0000 2026-02-13T02:06:22.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:06:22.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:06:22.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:06:22.967 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:06:22.967 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:06:22.967 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:06:22.967 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:06:22.967 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:06:23.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:06:04.033487+0000 '>' 2026-02-13T02:06:04.033487+0000 2026-02-13T02:06:23.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:06:24.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:06:24.136 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:06:24.137 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:06:24.137 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:06:24.137 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:06:24.137 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:06:24.137 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:06:24.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:06:04.033487+0000 '>' 2026-02-13T02:06:04.033487+0000 2026-02-13T02:06:24.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:06:25.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:06:25.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:06:25.302 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:06:25.302 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:06:25.302 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:06:25.303 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:06:25.303 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:06:25.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:06:23.078951+0000 '>' 2026-02-13T02:06:04.033487+0000 2026-02-13T02:06:25.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:06:25.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-13T02:06:25.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:06:25.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:06:25.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2026-02-13T02:06:25.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:06:25.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-13T02:06:25.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:06:25.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:06:25.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2026-02-13T02:06:25.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:06:25.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2026-02-13T02:06:25.471 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:06:25.471 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:06:25.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:06:25.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:06:25.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:06:25.779 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:06:25.779 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-13T02:06:25.779 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:06:25.779 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:06:25.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2026-02-13T02:06:25.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:06:25.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2026-02-13T02:06:25.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2026-02-13T02:06:25.864 INFO:tasks.workunit.client.0.trial112.stdout:_ 2026-02-13T02:06:25.864 INFO:tasks.workunit.client.0.trial112.stdout:hinfo_key 2026-02-13T02:06:25.864 INFO:tasks.workunit.client.0.trial112.stdout:snapset 2026-02-13T02:06:25.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2026-02-13T02:06:25.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:06:25.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:06:25.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2026-02-13T02:06:25.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:06:25.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2026-02-13T02:06:25.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:06:25.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:06:25.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:06:25.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:06:25.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2026-02-13T02:06:25.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2026-02-13T02:06:25.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:06:25.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:06:25.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:06:25.966 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:06:25.966 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:06:25.966 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:06:25.966 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:06:25.966 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:06:25.966 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:06:25.966 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:06:25.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:06:25.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:06:25.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:06:25.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:06:25.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:06:25.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:06:25.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:06:25.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:06:25.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:06:25.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:06:25.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2026-02-13T02:06:25.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2026-02-13T02:06:25.968 INFO:tasks.workunit.client.0.trial112.stderr:start osd.9 2026-02-13T02:06:25.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:06:25.968 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2026-02-13T02:06:25.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2026-02-13T02:06:25.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:06:25.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:06:25.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:06:25.984 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:06:25.972+0000 7fa82de8ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:06:25.986 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:06:25.976+0000 7fa82de8ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:06:25.987 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:06:25.976+0000 7fa82de8ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:06:26.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2026-02-13T02:06:26.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:06:26.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2026-02-13T02:06:26.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:06:26.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:06:26.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:06:26.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:06:26.134 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:06:26.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:06:26.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-13T02:06:26.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:06:26.359 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:06:26.348+0000 7fa82de8ea40 -1 Falling back to public interface 2026-02-13T02:06:26.985 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:06:26.972+0000 7fa82de8ea40 -1 osd.9 80 log_to_monitors true 2026-02-13T02:06:27.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:06:27.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:06:27.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:06:27.300 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:06:27.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:06:27.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-13T02:06:27.461 INFO:tasks.workunit.client.0.trial112.stderr:osd.9 up in weight 1 up_from 84 up_thru 0 down_at 81 last_clean_interval [72,80) [v2:127.0.0.1:6874/3896847709,v1:127.0.0.1:6875/3896847709] [v2:127.0.0.1:6876/3896847709,v1:127.0.0.1:6877/3896847709] exists,up 31da8501-3453-451f-af71-20390baae5a0 2026-02-13T02:06:27.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:06:27.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:06:27.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:06:27.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:06:27.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:06:27.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:06:27.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:06:27.462 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:06:27.463 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:06:27.463 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:06:27.463 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:06:27.463 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:06:27.463 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:06:27.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:06:27.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:06:27.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:06:27.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:06:27.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:06:27.579 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:06:27.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:06:27.740 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:06:27.740 INFO:tasks.workunit.client.0.trial112.stderr:2 2026-02-13T02:06:27.740 INFO:tasks.workunit.client.0.trial112.stderr:3 2026-02-13T02:06:27.740 INFO:tasks.workunit.client.0.trial112.stderr:4 2026-02-13T02:06:27.740 INFO:tasks.workunit.client.0.trial112.stderr:5 2026-02-13T02:06:27.741 INFO:tasks.workunit.client.0.trial112.stderr:6 2026-02-13T02:06:27.741 INFO:tasks.workunit.client.0.trial112.stderr:7 2026-02-13T02:06:27.741 INFO:tasks.workunit.client.0.trial112.stderr:8 2026-02-13T02:06:27.741 INFO:tasks.workunit.client.0.trial112.stderr:9' 2026-02-13T02:06:27.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:06:27.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:27.741 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:06:27.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836502 2026-02-13T02:06:27.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836502 2026-02-13T02:06:27.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502' 2026-02-13T02:06:27.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:27.813 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:06:27.886 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705685 2026-02-13T02:06:27.886 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705685 2026-02-13T02:06:27.886 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-38654705685' 2026-02-13T02:06:27.886 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:27.887 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:06:27.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574869 2026-02-13T02:06:27.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574869 2026-02-13T02:06:27.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-38654705685 2-55834574869' 2026-02-13T02:06:27.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:27.961 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:06:28.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=283467841546 2026-02-13T02:06:28.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 283467841546 2026-02-13T02:06:28.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-38654705685 2-55834574869 3-283467841546' 2026-02-13T02:06:28.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:28.033 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-13T02:06:28.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280531 2026-02-13T02:06:28.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280531 2026-02-13T02:06:28.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-38654705685 2-55834574869 3-283467841546 4-94489280531' 2026-02-13T02:06:28.107 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:28.107 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-13T02:06:28.181 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=335007449093 2026-02-13T02:06:28.181 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 335007449093 2026-02-13T02:06:28.181 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-38654705685 2-55834574869 3-283467841546 4-94489280531 5-335007449093' 2026-02-13T02:06:28.181 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:28.181 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-13T02:06:28.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953490 2026-02-13T02:06:28.253 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953490 2026-02-13T02:06:28.253 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-38654705685 2-55834574869 3-283467841546 4-94489280531 5-335007449093 6-137438953490' 2026-02-13T02:06:28.253 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:28.253 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-13T02:06:28.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789969 2026-02-13T02:06:28.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789969 2026-02-13T02:06:28.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-38654705685 2-55834574869 3-283467841546 4-94489280531 5-335007449093 6-137438953490 7-158913789969' 2026-02-13T02:06:28.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:28.327 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-13T02:06:28.402 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626448 2026-02-13T02:06:28.403 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626448 2026-02-13T02:06:28.403 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-38654705685 2-55834574869 3-283467841546 4-94489280531 5-335007449093 6-137438953490 7-158913789969 8-180388626448' 2026-02-13T02:06:28.403 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:28.403 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-13T02:06:28.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=360777252866 2026-02-13T02:06:28.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 360777252866 2026-02-13T02:06:28.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-38654705685 2-55834574869 3-283467841546 4-94489280531 5-335007449093 6-137438953490 7-158913789969 8-180388626448 9-360777252866' 2026-02-13T02:06:28.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:28.478 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836502 2026-02-13T02:06:28.479 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:28.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:06:28.480 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836502 2026-02-13T02:06:28.481 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:28.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836502 2026-02-13T02:06:28.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2026-02-13T02:06:28.482 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 21474836502 2026-02-13T02:06:28.482 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:06:28.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836501 -lt 21474836502 2026-02-13T02:06:28.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:06:29.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:06:29.649 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:06:29.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836502 -lt 21474836502 2026-02-13T02:06:29.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:29.809 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705685 2026-02-13T02:06:29.809 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:29.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:06:29.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705685 2026-02-13T02:06:29.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:29.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705685 2026-02-13T02:06:29.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705685' 2026-02-13T02:06:29.812 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 38654705685 2026-02-13T02:06:29.813 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:06:29.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705686 -lt 38654705685 2026-02-13T02:06:29.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:29.979 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574869 2026-02-13T02:06:29.980 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:29.981 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:06:29.981 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574869 2026-02-13T02:06:29.982 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:29.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574869 2026-02-13T02:06:29.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574869' 2026-02-13T02:06:29.983 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 55834574869 2026-02-13T02:06:29.984 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:06:30.155 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574869 -lt 55834574869 2026-02-13T02:06:30.155 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:30.156 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-283467841546 2026-02-13T02:06:30.156 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:30.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:06:30.158 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-283467841546 2026-02-13T02:06:30.158 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:30.160 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=283467841546 2026-02-13T02:06:30.160 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 283467841546' 2026-02-13T02:06:30.160 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.3 seq 283467841546 2026-02-13T02:06:30.160 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:06:30.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 283467841547 -lt 283467841546 2026-02-13T02:06:30.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:30.329 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-94489280531 2026-02-13T02:06:30.329 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:30.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-13T02:06:30.331 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-94489280531 2026-02-13T02:06:30.331 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:30.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280531 2026-02-13T02:06:30.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 94489280531' 2026-02-13T02:06:30.333 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.4 seq 94489280531 2026-02-13T02:06:30.333 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-13T02:06:30.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280531 -lt 94489280531 2026-02-13T02:06:30.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:30.490 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-335007449093 2026-02-13T02:06:30.491 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:30.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-13T02:06:30.493 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-335007449093 2026-02-13T02:06:30.493 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:30.494 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=335007449093 2026-02-13T02:06:30.494 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 335007449093' 2026-02-13T02:06:30.494 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.5 seq 335007449093 2026-02-13T02:06:30.495 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-13T02:06:30.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 335007449093 -lt 335007449093 2026-02-13T02:06:30.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:30.656 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953490 2026-02-13T02:06:30.656 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:30.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-13T02:06:30.658 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953490 2026-02-13T02:06:30.658 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:30.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953490 2026-02-13T02:06:30.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953490' 2026-02-13T02:06:30.660 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.6 seq 137438953490 2026-02-13T02:06:30.660 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-13T02:06:30.827 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953490 -lt 137438953490 2026-02-13T02:06:30.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:30.828 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789969 2026-02-13T02:06:30.828 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:30.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-13T02:06:30.830 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789969 2026-02-13T02:06:30.830 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:30.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789969 2026-02-13T02:06:30.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789969' 2026-02-13T02:06:30.832 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.7 seq 158913789969 2026-02-13T02:06:30.832 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-13T02:06:31.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789969 -lt 158913789969 2026-02-13T02:06:31.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:31.006 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626448 2026-02-13T02:06:31.006 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:31.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-13T02:06:31.008 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626448 2026-02-13T02:06:31.008 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:31.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626448 2026-02-13T02:06:31.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626448' 2026-02-13T02:06:31.009 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.8 seq 180388626448 2026-02-13T02:06:31.010 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-13T02:06:31.182 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626448 -lt 180388626448 2026-02-13T02:06:31.182 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:31.183 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-360777252866 2026-02-13T02:06:31.183 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:31.184 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-13T02:06:31.185 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-360777252866 2026-02-13T02:06:31.185 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:31.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=360777252866 2026-02-13T02:06:31.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 360777252866' 2026-02-13T02:06:31.187 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.9 seq 360777252866 2026-02-13T02:06:31.187 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-13T02:06:31.348 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 360777252866 -lt 360777252866 2026-02-13T02:06:31.349 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:06:31.349 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:06:31.349 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:06:31.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:06:31.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:06:31.547 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:06:31.548 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:06:31.548 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:06:31.548 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:06:31.548 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:06:31.548 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:06:31.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:06:31.704 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:06:31.704 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:06:31.704 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:06:31.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:06:31.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:06:31.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:06:31.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-13T02:06:31.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-13T02:06:31.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:322: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2026-02-13T02:06:31.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:245: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2026-02-13T02:06:31.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:246: corrupt_and_repair_two: local poolname=ecpool 2026-02-13T02:06:31.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:247: corrupt_and_repair_two: local first=5 2026-02-13T02:06:31.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_two: local second=9 2026-02-13T02:06:31.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_two: pids= 2026-02-13T02:06:31.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2026-02-13T02:06:31.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-13T02:06:31.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-13T02:06:31.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 182504"' 2026-02-13T02:06:31.920 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 182504' 2026-02-13T02:06:31.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2026-02-13T02:06:31.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-13T02:06:31.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-13T02:06:31.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 182506"' 2026-02-13T02:06:31.921 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 182506' 2026-02-13T02:06:31.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2026-02-13T02:06:31.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_two: wait_background pids 2026-02-13T02:06:31.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 182504 182506' 2026-02-13T02:06:31.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-13T02:06:31.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-13T02:06:31.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 182504 2026-02-13T02:06:31.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/182507: /' 2026-02-13T02:06:31.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2026-02-13T02:06:31.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/182510: /' 2026-02-13T02:06:32.530 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:06:32.530 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:06:32.530 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2026-02-13T02:06:32.530 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:06:32.531 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2026-02-13T02:06:32.531 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:06:32.531 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:06:32.531 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2026-02-13T02:06:32.531 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:06:32.531 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2026-02-13T02:06:32.531 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:06:32.531 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:06:32.531 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:06:32.531 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:06:32.531 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:06:32.531 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:06:32.532 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2026-02-13T02:06:32.532 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:06:32.532 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:06:32.532 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2026-02-13T02:06:32.532 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:06:32.532 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2026-02-13T02:06:32.532 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2026-02-13T02:06:32.532 INFO:tasks.workunit.client.0.trial112.stderr:182507: remove 1#2:eb822e21:::SOMETHING:head# 2026-02-13T02:06:32.532 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2026-02-13T02:06:32.532 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:06:32.532 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:06:32.532 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2026-02-13T02:06:32.533 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:06:32.533 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2026-02-13T02:06:32.533 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:06:32.533 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:06:32.534 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:06:32.534 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:06:32.534 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2026-02-13T02:06:32.534 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2026-02-13T02:06:32.534 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:06:32.535 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:06:32.535 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:06:32.535 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:06:32.535 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:06:32.535 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:06:32.535 INFO:tasks.workunit.client.0.trial112.stderr:182507: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:06:32.535 INFO:tasks.workunit.client.0.trial112.stderr:182507: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:06:32.535 INFO:tasks.workunit.client.0.trial112.stderr:182507: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:06:32.535 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:06:32.535 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:06:32.535 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:06:32.535 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:06:32.535 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:06:32.536 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:06:32.536 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:06:32.536 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:06:32.536 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:06:32.536 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:06:32.536 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:06:32.536 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2026-02-13T02:06:32.536 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2026-02-13T02:06:32.536 INFO:tasks.workunit.client.0.trial112.stderr:182507: start osd.5 2026-02-13T02:06:32.536 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:06:32.590 INFO:tasks.workunit.client.0.trial112.stderr:1182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:06:32.590 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:06:32.590 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2026-02-13T02:06:32.590 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:06:32.591 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2026-02-13T02:06:32.591 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:06:32.591 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:06:32.591 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2026-02-13T02:06:32.591 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:06:32.591 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2026-02-13T02:06:32.591 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:06:32.591 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:06:32.591 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:06:32.591 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:06:32.591 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:06:32.591 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:06:32.592 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2026-02-13T02:06:32.592 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:06:32.592 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:06:32.592 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2026-02-13T02:06:32.592 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:06:32.592 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2026-02-13T02:06:32.592 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2026-02-13T02:06:32.592 INFO:tasks.workunit.client.0.trial112.stderr:182510: remove 2#2:eb822e21:::SOMETHING:head# 2026-02-13T02:06:32.592 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2026-02-13T02:06:32.592 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:06:32.592 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:06:32.592 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2026-02-13T02:06:32.592 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:06:32.593 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2026-02-13T02:06:32.593 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:06:32.593 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:06:32.593 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:06:32.594 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:06:32.594 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2026-02-13T02:06:32.594 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2026-02-13T02:06:32.594 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:06:32.594 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:06:32.594 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:06:32.594 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:06:32.594 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:06:32.594 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:06:32.594 INFO:tasks.workunit.client.0.trial112.stderr:182510: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:06:32.594 INFO:tasks.workunit.client.0.trial112.stderr:182510: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:06:32.595 INFO:tasks.workunit.client.0.trial112.stderr:182510: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:06:32.595 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:06:32.595 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:06:32.595 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:06:32.595 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:06:32.595 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:06:32.595 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:06:32.595 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:06:32.595 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:06:32.595 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:06:32.595 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:06:32.595 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:06:32.595 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2026-02-13T02:06:32.596 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2026-02-13T02:06:32.596 INFO:tasks.workunit.client.0.trial112.stderr:182510: start osd.9 2026-02-13T02:06:32.596 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:06:34.031 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2026-02-13T02:06:34.031 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2026-02-13T02:06:34.031 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:06:34.031 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:06:34.032 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:06:34.032 INFO:tasks.workunit.client.0.trial112.stderr:182507: 2026-02-13T02:06:32.544+0000 7f95c20bba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:06:34.032 INFO:tasks.workunit.client.0.trial112.stderr:182507: 2026-02-13T02:06:32.544+0000 7f95c20bba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:06:34.032 INFO:tasks.workunit.client.0.trial112.stderr:182507: 2026-02-13T02:06:32.544+0000 7f95c20bba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:06:34.032 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2026-02-13T02:06:34.032 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:06:34.032 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-13T02:06:34.032 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:06:34.032 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:06:34.032 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:06:34.032 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:06:34.032 INFO:tasks.workunit.client.0.trial112.stderr:182507: 0 2026-02-13T02:06:34.033 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:06:34.033 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-13T02:06:34.033 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:06:34.033 INFO:tasks.workunit.client.0.trial112.stderr:182507: 2026-02-13T02:06:32.992+0000 7f95c20bba40 -1 Falling back to public interface 2026-02-13T02:06:34.033 INFO:tasks.workunit.client.0.trial112.stderr:182507: 2026-02-13T02:06:33.628+0000 7f95c20bba40 -1 osd.5 85 log_to_monitors true 2026-02-13T02:06:34.033 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:06:34.033 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:06:34.033 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:06:34.033 INFO:tasks.workunit.client.0.trial112.stderr:182507: 1 2026-02-13T02:06:34.033 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:06:34.033 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-13T02:06:34.034 INFO:tasks.workunit.client.0.trial112.stderr:182507: osd.5 up in weight 1 up_from 89 up_thru 89 down_at 86 last_clean_interval [78,85) [v2:127.0.0.1:6842/2995304824,v1:127.0.0.1:6843/2995304824] [v2:127.0.0.1:6844/2995304824,v1:127.0.0.1:6845/2995304824] exists,up 670b1f5b-3687-4794-8a8a-767a3a2770e7 2026-02-13T02:06:34.034 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:06:34.034 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:06:34.034 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:06:34.034 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:06:34.034 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:06:34.034 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:06:34.034 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:06:34.035 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:06:34.035 INFO:tasks.workunit.client.0.trial112.stderr:182507: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:06:34.089 INFO:tasks.workunit.client.0.trial112.stderr:182507: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: g82510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2026-02-13T02:06:34.089 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2026-02-13T02:06:34.090 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:06:34.090 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:06:34.090 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:06:34.090 INFO:tasks.workunit.client.0.trial112.stderr:182510: 2026-02-13T02:06:32.600+0000 7fd284852a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:06:34.090 INFO:tasks.workunit.client.0.trial112.stderr:182510: 2026-02-13T02:06:32.604+0000 7fd284852a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:06:34.090 INFO:tasks.workunit.client.0.trial112.stderr:182510: 2026-02-13T02:06:32.604+0000 7fd284852a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:06:34.090 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2026-02-13T02:06:34.090 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:06:34.090 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2026-02-13T02:06:34.090 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:06:34.090 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:06:34.090 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:06:34.091 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:06:34.091 INFO:tasks.workunit.client.0.trial112.stderr:182510: 0 2026-02-13T02:06:34.091 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:06:34.091 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-13T02:06:34.091 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:06:34.091 INFO:tasks.workunit.client.0.trial112.stderr:182510: 2026-02-13T02:06:33.068+0000 7fd284852a40 -1 Falling back to public interface 2026-02-13T02:06:34.091 INFO:tasks.workunit.client.0.trial112.stderr:182510: 2026-02-13T02:06:33.728+0000 7fd284852a40 -1 osd.9 85 log_to_monitors true 2026-02-13T02:06:34.091 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:06:34.091 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:06:34.091 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:06:34.091 INFO:tasks.workunit.client.0.trial112.stderr:182510: 1 2026-02-13T02:06:34.091 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:06:34.092 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-13T02:06:34.092 INFO:tasks.workunit.client.0.trial112.stderr:182510: osd.9 up in weight 1 up_from 90 up_thru 0 down_at 86 last_clean_interval [84,85) [v2:127.0.0.1:6874/349573942,v1:127.0.0.1:6875/349573942] [v2:127.0.0.1:6876/349573942,v1:127.0.0.1:6877/349573942] exists,up 31da8501-3453-451f-af71-20390baae5a0 2026-02-13T02:06:34.092 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:06:34.092 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:06:34.092 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:06:34.092 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:06:34.092 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:06:34.092 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:06:34.092 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:06:34.092 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:06:34.092 INFO:tasks.workunit.client.0.trial112.stderr:182510: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:06:34.591 INFO:tasks.workunit.client.0.trial112.stderr:182510: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_tiet_timeout_delays: echo true 2026-02-13T02:06:34.591 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:06:34.591 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:06:34.591 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:06:34.591 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:06:34.591 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:06:34.592 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:06:34.592 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:06:34.592 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:06:34.592 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:06:34.592 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:06:34.592 INFO:tasks.workunit.client.0.trial112.stderr:182507: 1 2026-02-13T02:06:34.592 INFO:tasks.workunit.client.0.trial112.stderr:182507: 2 2026-02-13T02:06:34.592 INFO:tasks.workunit.client.0.trial112.stderr:182507: 3 2026-02-13T02:06:34.592 INFO:tasks.workunit.client.0.trial112.stderr:182507: 4 2026-02-13T02:06:34.592 INFO:tasks.workunit.client.0.trial112.stderr:182507: 5 2026-02-13T02:06:34.592 INFO:tasks.workunit.client.0.trial112.stderr:182507: 6 2026-02-13T02:06:34.592 INFO:tasks.workunit.client.0.trial112.stderr:182507: 7 2026-02-13T02:06:34.593 INFO:tasks.workunit.client.0.trial112.stderr:182507: 8 2026-02-13T02:06:34.593 INFO:tasks.workunit.client.0.trial112.stderr:182507: 9' 2026-02-13T02:06:34.593 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:06:34.593 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:34.593 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:06:34.593 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836504 2026-02-13T02:06:34.593 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836504 2026-02-13T02:06:34.593 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504' 2026-02-13T02:06:34.593 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:34.593 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:06:34.593 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705688 2026-02-13T02:06:34.593 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705688 2026-02-13T02:06:34.594 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-38654705688' 2026-02-13T02:06:34.594 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:34.594 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:06:34.594 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574871 2026-02-13T02:06:34.594 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574871 2026-02-13T02:06:34.594 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-38654705688 2-55834574871' 2026-02-13T02:06:34.594 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:34.594 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:06:34.594 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=283467841549 2026-02-13T02:06:34.594 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 283467841549 2026-02-13T02:06:34.594 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-38654705688 2-55834574871 3-283467841549' 2026-02-13T02:06:34.594 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:34.662 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258:meout_delays: echo true 2026-02-13T02:06:34.662 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:06:34.662 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:06:34.662 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:06:34.663 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:06:34.663 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:06:34.663 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:06:34.663 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:06:34.663 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:06:34.663 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:06:34.663 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:06:34.663 INFO:tasks.workunit.client.0.trial112.stderr:182510: 1 2026-02-13T02:06:34.664 INFO:tasks.workunit.client.0.trial112.stderr:182510: 2 2026-02-13T02:06:34.664 INFO:tasks.workunit.client.0.trial112.stderr:182510: 3 2026-02-13T02:06:34.664 INFO:tasks.workunit.client.0.trial112.stderr:182510: 4 2026-02-13T02:06:34.664 INFO:tasks.workunit.client.0.trial112.stderr:182510: 5 2026-02-13T02:06:34.664 INFO:tasks.workunit.client.0.trial112.stderr:182510: 6 2026-02-13T02:06:34.664 INFO:tasks.workunit.client.0.trial112.stderr:182510: 7 2026-02-13T02:06:34.664 INFO:tasks.workunit.client.0.trial112.stderr:182510: 8 2026-02-13T02:06:34.664 INFO:tasks.workunit.client.0.trial112.stderr:182510: 9' 2026-02-13T02:06:34.664 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:06:34.664 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:34.664 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:06:34.664 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836505 2026-02-13T02:06:34.665 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836505 2026-02-13T02:06:34.665 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505' 2026-02-13T02:06:34.665 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:34.665 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:06:34.665 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705689 2026-02-13T02:06:34.665 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705689 2026-02-13T02:06:34.665 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-38654705689' 2026-02-13T02:06:34.665 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:34.665 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:06:34.665 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574872 2026-02-13T02:06:34.665 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574872 2026-02-13T02:06:34.665 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-38654705689 2-55834574872' 2026-02-13T02:06:34.665 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:34.666 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:06:34.666 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=283467841550 2026-02-13T02:06:34.666 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 283467841550 2026-02-13T02:06:34.666 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-38654705689 2-55834574872 3-283467841550' 2026-02-13T02:06:34.666 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:35.032 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flus flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-13T02:06:35.032 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280533 2026-02-13T02:06:35.033 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280533 2026-02-13T02:06:35.033 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-38654705688 2-55834574871 3-283467841549 4-94489280533' 2026-02-13T02:06:35.033 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:35.033 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-13T02:06:35.033 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=382252089346 2026-02-13T02:06:35.033 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 382252089346 2026-02-13T02:06:35.033 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-38654705688 2-55834574871 3-283467841549 4-94489280533 5-382252089346' 2026-02-13T02:06:35.033 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:35.033 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-13T02:06:35.033 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953492 2026-02-13T02:06:35.033 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953492 2026-02-13T02:06:35.034 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-38654705688 2-55834574871 3-283467841549 4-94489280533 5-382252089346 6-137438953492' 2026-02-13T02:06:35.034 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:35.034 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-13T02:06:35.034 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789971 2026-02-13T02:06:35.034 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789971 2026-02-13T02:06:35.034 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-38654705688 2-55834574871 3-283467841549 4-94489280533 5-382252089346 6-137438953492 7-158913789971' 2026-02-13T02:06:35.034 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:35.034 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-13T02:06:35.034 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626451 2026-02-13T02:06:35.034 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626451 2026-02-13T02:06:35.034 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-38654705688 2-55834574871 3-283467841549 4-94489280533 5-382252089346 6-137438953492 7-158913789971 8-180388626451' 2026-02-13T02:06:35.034 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:35.035 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-13T02:06:35.035 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=386547056642 2026-02-13T02:06:35.035 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 386547056642 2026-02-13T02:06:35.035 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-38654705688 2-55834574871 3-283467841549 4-94489280533 5-382252089346 6-137438953492 7-158913789971 8-180388626451 9-386547056642' 2026-02-13T02:06:35.035 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:35.103 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/ceph_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-13T02:06:35.103 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280534 2026-02-13T02:06:35.103 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280534 2026-02-13T02:06:35.103 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-38654705689 2-55834574872 3-283467841550 4-94489280534' 2026-02-13T02:06:35.103 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:35.103 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-13T02:06:35.104 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=382252089347 2026-02-13T02:06:35.104 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 382252089347 2026-02-13T02:06:35.104 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-38654705689 2-55834574872 3-283467841550 4-94489280534 5-382252089347' 2026-02-13T02:06:35.104 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:35.104 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-13T02:06:35.104 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953493 2026-02-13T02:06:35.104 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953493 2026-02-13T02:06:35.104 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-38654705689 2-55834574872 3-283467841550 4-94489280534 5-382252089347 6-137438953493' 2026-02-13T02:06:35.104 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:35.104 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-13T02:06:35.104 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789972 2026-02-13T02:06:35.105 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789972 2026-02-13T02:06:35.105 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-38654705689 2-55834574872 3-283467841550 4-94489280534 5-382252089347 6-137438953493 7-158913789972' 2026-02-13T02:06:35.105 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:35.105 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-13T02:06:35.105 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626452 2026-02-13T02:06:35.105 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626452 2026-02-13T02:06:35.105 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-38654705689 2-55834574872 3-283467841550 4-94489280534 5-382252089347 6-137438953493 7-158913789972 8-180388626452' 2026-02-13T02:06:35.105 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:35.105 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-13T02:06:35.105 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=386547056643 2026-02-13T02:06:35.105 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 386547056643 2026-02-13T02:06:35.105 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-38654705689 2-55834574872 3-283467841550 4-94489280534 5-382252089347 6-137438953493 7-158913789972 8-180388626452 9-386547056643' 2026-02-13T02:06:35.106 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:36.705 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtesthtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836504 2026-02-13T02:06:36.705 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:36.705 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:06:36.705 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836504 2026-02-13T02:06:36.706 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:36.706 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836504 2026-02-13T02:06:36.706 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2026-02-13T02:06:36.706 INFO:tasks.workunit.client.0.trial112.stderr:182507: waiting osd.0 seq 21474836504 2026-02-13T02:06:36.706 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:06:36.706 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836503 -lt 21474836504 2026-02-13T02:06:36.706 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:06:36.706 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:06:36.706 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:06:36.706 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836505 -lt 21474836504 2026-02-13T02:06:36.707 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:36.707 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705688 2026-02-13T02:06:36.707 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:36.707 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:06:36.707 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705688 2026-02-13T02:06:36.707 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:36.707 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705688 2026-02-13T02:06:36.707 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705688' 2026-02-13T02:06:36.707 INFO:tasks.workunit.client.0.trial112.stderr:182507: waiting osd.1 seq 38654705688 2026-02-13T02:06:36.707 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:06:36.708 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705689 -lt 38654705688 2026-02-13T02:06:36.708 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:36.708 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574871 2026-02-13T02:06:36.708 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:36.708 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:06:36.708 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574871 2026-02-13T02:06:36.708 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:36.708 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574871 2026-02-13T02:06:36.708 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574871' 2026-02-13T02:06:36.708 INFO:tasks.workunit.client.0.trial112.stderr:182507: waiting osd.2 seq 55834574871 2026-02-13T02:06:36.708 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:06:36.709 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574872 -lt 55834574871 2026-02-13T02:06:36.784 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stat/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836505 2026-02-13T02:06:36.784 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:36.784 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:06:36.784 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836505 2026-02-13T02:06:36.784 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:36.784 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836505 2026-02-13T02:06:36.785 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2026-02-13T02:06:36.785 INFO:tasks.workunit.client.0.trial112.stderr:182510: waiting osd.0 seq 21474836505 2026-02-13T02:06:36.785 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:06:36.785 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836503 -lt 21474836505 2026-02-13T02:06:36.785 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:06:36.785 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:06:36.785 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:06:36.785 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836505 -lt 21474836505 2026-02-13T02:06:36.785 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:36.785 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705689 2026-02-13T02:06:36.785 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:36.785 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:06:36.785 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705689 2026-02-13T02:06:36.785 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:36.786 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705689 2026-02-13T02:06:36.786 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705689' 2026-02-13T02:06:36.786 INFO:tasks.workunit.client.0.trial112.stderr:182510: waiting osd.1 seq 38654705689 2026-02-13T02:06:36.786 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:06:36.786 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705689 -lt 38654705689 2026-02-13T02:06:36.786 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:36.786 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574872 2026-02-13T02:06:36.786 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:36.786 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:06:36.786 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574872 2026-02-13T02:06:36.786 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:36.786 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574872 2026-02-13T02:06:36.786 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574872' 2026-02-13T02:06:36.786 INFO:tasks.workunit.client.0.trial112.stderr:182510: waiting osd.2 seq 55834574872 2026-02-13T02:06:36.786 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:06:36.786 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574872 -lt 55834574872 2026-02-13T02:06:37.206 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: fs: for s in $seqs 2026-02-13T02:06:37.206 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-283467841549 2026-02-13T02:06:37.206 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:37.206 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:06:37.206 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-283467841549 2026-02-13T02:06:37.206 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:37.207 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=283467841549 2026-02-13T02:06:37.207 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 283467841549' 2026-02-13T02:06:37.207 INFO:tasks.workunit.client.0.trial112.stderr:182507: waiting osd.3 seq 283467841549 2026-02-13T02:06:37.207 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:06:37.207 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 283467841550 -lt 283467841549 2026-02-13T02:06:37.207 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:37.207 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-94489280533 2026-02-13T02:06:37.207 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:37.207 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-13T02:06:37.207 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-94489280533 2026-02-13T02:06:37.207 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:37.207 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280533 2026-02-13T02:06:37.208 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 94489280533' 2026-02-13T02:06:37.208 INFO:tasks.workunit.client.0.trial112.stderr:182507: waiting osd.4 seq 94489280533 2026-02-13T02:06:37.208 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-13T02:06:37.208 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280535 -lt 94489280533 2026-02-13T02:06:37.208 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:37.208 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-382252089346 2026-02-13T02:06:37.208 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:37.208 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-13T02:06:37.208 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-382252089346 2026-02-13T02:06:37.208 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:37.208 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=382252089346 2026-02-13T02:06:37.208 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 382252089346' 2026-02-13T02:06:37.209 INFO:tasks.workunit.client.0.trial112.stderr:182507: waiting osd.5 seq 382252089346 2026-02-13T02:06:37.209 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-13T02:06:37.209 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 382252089347 -lt 382252089346 2026-02-13T02:06:37.209 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:37.209 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953492 2026-02-13T02:06:37.209 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:37.209 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-13T02:06:37.295 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/cephor s in $seqs 2026-02-13T02:06:37.295 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-283467841550 2026-02-13T02:06:37.295 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:37.295 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:06:37.295 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-283467841550 2026-02-13T02:06:37.295 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:37.295 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=283467841550 2026-02-13T02:06:37.296 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 283467841550' 2026-02-13T02:06:37.296 INFO:tasks.workunit.client.0.trial112.stderr:182510: waiting osd.3 seq 283467841550 2026-02-13T02:06:37.296 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:06:37.296 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 283467841550 -lt 283467841550 2026-02-13T02:06:37.296 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:37.296 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-94489280534 2026-02-13T02:06:37.296 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:37.296 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-13T02:06:37.296 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-94489280534 2026-02-13T02:06:37.296 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:37.296 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280534 2026-02-13T02:06:37.297 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 94489280534' 2026-02-13T02:06:37.297 INFO:tasks.workunit.client.0.trial112.stderr:182510: waiting osd.4 seq 94489280534 2026-02-13T02:06:37.297 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-13T02:06:37.297 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280535 -lt 94489280534 2026-02-13T02:06:37.297 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:37.297 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-382252089347 2026-02-13T02:06:37.297 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:37.297 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-13T02:06:37.297 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-382252089347 2026-02-13T02:06:37.297 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:37.297 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=382252089347 2026-02-13T02:06:37.297 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 382252089347' 2026-02-13T02:06:37.297 INFO:tasks.workunit.client.0.trial112.stderr:182510: waiting osd.5 seq 382252089347 2026-02-13T02:06:37.298 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-13T02:06:37.298 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 382252089347 -lt 382252089347 2026-02-13T02:06:37.298 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:37.298 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953493 2026-02-13T02:06:37.298 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:37.298 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-13T02:06:37.703 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-help-helpers.sh:2268: flush_pg_stats: echo 6-137438953492 2026-02-13T02:06:37.703 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:37.703 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953492 2026-02-13T02:06:37.704 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953492' 2026-02-13T02:06:37.704 INFO:tasks.workunit.client.0.trial112.stderr:182507: waiting osd.6 seq 137438953492 2026-02-13T02:06:37.704 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-13T02:06:37.704 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953494 -lt 137438953492 2026-02-13T02:06:37.704 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:37.704 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789971 2026-02-13T02:06:37.704 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:37.704 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-13T02:06:37.704 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789971 2026-02-13T02:06:37.704 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:37.704 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789971 2026-02-13T02:06:37.705 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789971' 2026-02-13T02:06:37.705 INFO:tasks.workunit.client.0.trial112.stderr:182507: waiting osd.7 seq 158913789971 2026-02-13T02:06:37.705 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-13T02:06:37.705 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789973 -lt 158913789971 2026-02-13T02:06:37.705 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:37.705 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626451 2026-02-13T02:06:37.705 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:37.705 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-13T02:06:37.705 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626451 2026-02-13T02:06:37.705 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:37.705 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626451 2026-02-13T02:06:37.705 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626451' 2026-02-13T02:06:37.706 INFO:tasks.workunit.client.0.trial112.stderr:182507: waiting osd.8 seq 180388626451 2026-02-13T02:06:37.706 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-13T02:06:37.706 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626452 -lt 180388626451 2026-02-13T02:06:37.706 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:37.706 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-386547056642 2026-02-13T02:06:37.706 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:37.706 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-13T02:06:37.706 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-386547056642 2026-02-13T02:06:37.706 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:37.706 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=386547056642 2026-02-13T02:06:37.809 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 3865470ers.sh:2268: flush_pg_stats: echo 6-137438953493 2026-02-13T02:06:37.809 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:37.809 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953493 2026-02-13T02:06:37.809 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953493' 2026-02-13T02:06:37.810 INFO:tasks.workunit.client.0.trial112.stderr:182510: waiting osd.6 seq 137438953493 2026-02-13T02:06:37.810 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-13T02:06:37.810 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953494 -lt 137438953493 2026-02-13T02:06:37.810 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:37.810 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789972 2026-02-13T02:06:37.810 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:37.810 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-13T02:06:37.810 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789972 2026-02-13T02:06:37.810 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:37.810 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789972 2026-02-13T02:06:37.810 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789972' 2026-02-13T02:06:37.811 INFO:tasks.workunit.client.0.trial112.stderr:182510: waiting osd.7 seq 158913789972 2026-02-13T02:06:37.811 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-13T02:06:37.811 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789973 -lt 158913789972 2026-02-13T02:06:37.811 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:37.811 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626452 2026-02-13T02:06:37.811 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:37.811 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-13T02:06:37.811 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626452 2026-02-13T02:06:37.811 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:37.811 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626452 2026-02-13T02:06:37.811 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626452' 2026-02-13T02:06:37.811 INFO:tasks.workunit.client.0.trial112.stderr:182510: waiting osd.8 seq 180388626452 2026-02-13T02:06:37.812 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-13T02:06:37.812 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626452 -lt 180388626452 2026-02-13T02:06:37.812 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:37.812 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-386547056643 2026-02-13T02:06:37.812 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:37.812 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-13T02:06:37.812 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-386547056643 2026-02-13T02:06:37.812 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:37.812 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=386547056643 2026-02-13T02:06:38.411 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 38654705664356642' 2026-02-13T02:06:38.411 INFO:tasks.workunit.client.0.trial112.stderr:182507: waiting osd.9 seq 386547056642 2026-02-13T02:06:38.411 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-13T02:06:38.411 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 386547056643 -lt 386547056642 2026-02-13T02:06:38.411 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:06:38.411 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:06:38.411 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:06:38.412 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:06:38.412 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:06:38.412 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:06:38.412 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:06:38.412 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:06:38.412 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:06:38.412 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:06:38.412 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:06:38.412 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:06:38.412 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:06:38.412 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:06:38.412 INFO:tasks.workunit.client.0.trial112.stderr:182507: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:06:38.413 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:06:38.413 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:06:38.413 INFO:tasks.workunit.client.0.trial112.stderr:182507: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:06:38.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-13T02:06:38.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-13T02:06:38.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 182506 2026-02-13T02:06:38.555 INFO:tasks.workunit.client.0.trial112.stderr:' 2026-02-13T02:06:38.555 INFO:tasks.workunit.client.0.trial112.stderr:182510: waiting osd.9 seq 386547056643 2026-02-13T02:06:38.555 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-13T02:06:38.556 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 386547056643 -lt 386547056643 2026-02-13T02:06:38.556 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:06:38.556 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:06:38.556 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:06:38.556 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:06:38.556 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:06:38.556 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:06:38.556 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:06:38.556 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:06:38.556 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:06:38.556 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:06:38.556 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:06:38.557 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:06:38.557 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:06:38.557 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:06:38.557 INFO:tasks.workunit.client.0.trial112.stderr:182510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:06:38.557 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:06:38.557 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:06:38.557 INFO:tasks.workunit.client.0.trial112.stderr:182510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:06:38.557 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-13T02:06:38.558 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-13T02:06:38.559 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-13T02:06:38.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-13T02:06:38.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:257: corrupt_and_repair_two: return_code=0 2026-02-13T02:06:38.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:258: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-13T02:06:38.559 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: get_pg ecpool SOMETHING 2026-02-13T02:06:38.559 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-13T02:06:38.559 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-13T02:06:38.559 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-13T02:06:38.559 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T02:06:38.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: local pg=2.0 2026-02-13T02:06:38.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:264: corrupt_and_repair_two: repair 2.0 2026-02-13T02:06:38.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-13T02:06:38.719 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-13T02:06:38.719 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:06:38.720 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:06:38.720 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:06:38.720 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:06:38.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-13T02:06:23.078951+0000 2026-02-13T02:06:38.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-13T02:06:39.032 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-13T02:06:39.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-13T02:06:23.078951+0000 2026-02-13T02:06:39.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-13T02:06:39.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:06:23.078951+0000 2026-02-13T02:06:39.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:06:39.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:06:39.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:06:39.043 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:06:39.044 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:06:39.044 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:06:39.044 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:06:39.044 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:06:39.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:06:23.078951+0000 '>' 2026-02-13T02:06:23.078951+0000 2026-02-13T02:06:39.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:06:40.206 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:06:40.206 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:06:40.207 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:06:40.207 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:06:40.207 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:06:40.208 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:06:40.208 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:06:40.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:06:23.078951+0000 '>' 2026-02-13T02:06:23.078951+0000 2026-02-13T02:06:40.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:06:41.368 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:06:41.368 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:06:41.370 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:06:41.370 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:06:41.370 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:06:41.370 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:06:41.370 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:06:41.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:06:23.078951+0000 '>' 2026-02-13T02:06:23.078951+0000 2026-02-13T02:06:41.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:06:42.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:06:42.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:06:42.532 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:06:42.533 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:06:42.533 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:06:42.533 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:06:42.533 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:06:42.692 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:06:23.078951+0000 '>' 2026-02-13T02:06:23.078951+0000 2026-02-13T02:06:42.692 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:06:43.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:06:43.694 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:06:43.695 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:06:43.695 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:06:43.695 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:06:43.695 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:06:43.695 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:06:43.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:06:23.078951+0000 '>' 2026-02-13T02:06:23.078951+0000 2026-02-13T02:06:43.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:06:44.854 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:06:44.854 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:06:44.855 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:06:44.855 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:06:44.856 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:06:44.856 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:06:44.856 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:06:45.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:06:23.078951+0000 '>' 2026-02-13T02:06:23.078951+0000 2026-02-13T02:06:45.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:06:46.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:06:46.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:06:46.015 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:06:46.015 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:06:46.015 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:06:46.016 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:06:46.016 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:06:46.178 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:06:39.990298+0000 '>' 2026-02-13T02:06:23.078951+0000 2026-02-13T02:06:46.178 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:06:46.178 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: corrupt_and_repair_two: pids= 2026-02-13T02:06:46.178 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-13T02:06:46.178 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-13T02:06:46.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-13T02:06:46.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 185966"' 2026-02-13T02:06:46.179 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 185966' 2026-02-13T02:06:46.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-13T02:06:46.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-13T02:06:46.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-13T02:06:46.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 185968"' 2026-02-13T02:06:46.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-13T02:06:46.179 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 185968' 2026-02-13T02:06:46.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:271: corrupt_and_repair_two: wait_background pids 2026-02-13T02:06:46.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 185966 185968' 2026-02-13T02:06:46.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-13T02:06:46.180 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-13T02:06:46.180 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 185966 2026-02-13T02:06:46.180 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/185969: /' 2026-02-13T02:06:46.180 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-13T02:06:46.180 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/185971: /' 2026-02-13T02:06:46.681 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:06:46.682 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:06:46.682 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2026-02-13T02:06:46.682 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:06:46.682 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-13T02:06:46.682 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:06:46.682 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:06:46.682 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2026-02-13T02:06:46.682 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:06:46.682 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2026-02-13T02:06:46.683 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:06:46.683 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:06:46.683 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:06:46.683 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:06:46.683 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:06:46.683 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:06:46.683 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-13T02:06:46.683 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:06:46.683 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:06:46.683 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2026-02-13T02:06:46.683 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:06:46.683 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2026-02-13T02:06:46.683 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2026-02-13T02:06:46.684 INFO:tasks.workunit.client.0.trial112.stderr:185971: _ 2026-02-13T02:06:46.684 INFO:tasks.workunit.client.0.trial112.stderr:185971: hinfo_key 2026-02-13T02:06:46.684 INFO:tasks.workunit.client.0.trial112.stderr:185971: snapset 2026-02-13T02:06:46.684 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2026-02-13T02:06:46.684 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:06:46.684 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:06:46.684 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2026-02-13T02:06:46.684 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:06:46.684 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2026-02-13T02:06:46.684 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:06:46.684 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:06:46.685 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:06:46.685 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:06:46.685 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2026-02-13T02:06:46.685 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2026-02-13T02:06:46.685 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:06:46.685 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:06:46.686 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:06:46.686 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:06:46.686 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:06:46.686 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:06:46.686 INFO:tasks.workunit.client.0.trial112.stderr:185971: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:06:46.686 INFO:tasks.workunit.client.0.trial112.stderr:185971: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:06:46.686 INFO:tasks.workunit.client.0.trial112.stderr:185971: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:06:46.686 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:06:46.686 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:06:46.686 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:06:46.686 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:06:46.686 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:06:46.687 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:06:46.687 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:06:46.687 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:06:46.687 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:06:46.687 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:06:46.687 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:06:46.687 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2026-02-13T02:06:46.687 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2026-02-13T02:06:46.687 INFO:tasks.workunit.client.0.trial112.stderr:185971: start osd.9 2026-02-13T02:06:46.698 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reco185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:06:46.698 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:06:46.698 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2026-02-13T02:06:46.698 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:06:46.698 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-13T02:06:46.698 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:06:46.698 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:06:46.698 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2026-02-13T02:06:46.698 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:06:46.699 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2026-02-13T02:06:46.699 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:06:46.699 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:06:46.699 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:06:46.699 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:06:46.699 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:06:46.699 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:06:46.699 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-13T02:06:46.699 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:06:46.699 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:06:46.699 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2026-02-13T02:06:46.699 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:06:46.699 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2026-02-13T02:06:46.700 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2026-02-13T02:06:46.700 INFO:tasks.workunit.client.0.trial112.stderr:185969: _ 2026-02-13T02:06:46.700 INFO:tasks.workunit.client.0.trial112.stderr:185969: hinfo_key 2026-02-13T02:06:46.700 INFO:tasks.workunit.client.0.trial112.stderr:185969: snapset 2026-02-13T02:06:46.700 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2026-02-13T02:06:46.700 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:06:46.700 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:06:46.700 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2026-02-13T02:06:46.700 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:06:46.700 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2026-02-13T02:06:46.700 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:06:46.700 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:06:46.701 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:06:46.701 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:06:46.701 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2026-02-13T02:06:46.701 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2026-02-13T02:06:46.701 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:06:46.701 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:06:46.701 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:06:46.702 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:06:46.702 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:06:46.702 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:06:46.702 INFO:tasks.workunit.client.0.trial112.stderr:185969: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:06:46.702 INFO:tasks.workunit.client.0.trial112.stderr:185969: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:06:46.702 INFO:tasks.workunit.client.0.trial112.stderr:185969: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:06:46.702 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:06:46.702 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:06:46.702 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:06:46.702 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:06:46.702 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:06:46.702 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:06:46.703 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:06:46.703 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:06:46.703 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:06:46.703 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:06:46.703 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:06:46.703 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2026-02-13T02:06:46.703 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2026-02-13T02:06:46.703 INFO:tasks.workunit.client.0.trial112.stderr:185969: start osd.5 2026-02-13T02:06:48.176 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:06:48.177 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2026-02-13T02:06:48.177 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2026-02-13T02:06:48.177 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:06:48.177 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:06:48.177 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:06:48.177 INFO:tasks.workunit.client.0.trial112.stderr:185971: 2026-02-13T02:06:46.692+0000 7f31fbd9ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:06:48.177 INFO:tasks.workunit.client.0.trial112.stderr:185971: 2026-02-13T02:06:46.692+0000 7f31fbd9ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:06:48.177 INFO:tasks.workunit.client.0.trial112.stderr:185971: 2026-02-13T02:06:46.692+0000 7f31fbd9ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:06:48.177 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2026-02-13T02:06:48.178 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:06:48.178 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2026-02-13T02:06:48.178 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:06:48.178 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:06:48.178 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:06:48.178 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:06:48.178 INFO:tasks.workunit.client.0.trial112.stderr:185971: 0 2026-02-13T02:06:48.178 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:06:48.178 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-13T02:06:48.178 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:06:48.178 INFO:tasks.workunit.client.0.trial112.stderr:185971: 2026-02-13T02:06:47.200+0000 7f31fbd9ea40 -1 Falling back to public interface 2026-02-13T02:06:48.178 INFO:tasks.workunit.client.0.trial112.stderr:185971: 2026-02-13T02:06:47.928+0000 7f31fbd9ea40 -1 osd.9 91 log_to_monitors true 2026-02-13T02:06:48.179 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:06:48.179 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:06:48.179 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:06:48.179 INFO:tasks.workunit.client.0.trial112.stderr:185971: 1 2026-02-13T02:06:48.179 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:06:48.179 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-13T02:06:48.179 INFO:tasks.workunit.client.0.trial112.stderr:185971: osd.9 up in weight 1 up_from 95 up_thru 0 down_at 92 last_clean_interval [90,91) [v2:127.0.0.1:6874/2322161612,v1:127.0.0.1:6875/2322161612] [v2:127.0.0.1:6876/2322161612,v1:127.0.0.1:6877/2322161612] exists,up 31da8501-3453-451f-af71-20390baae5a0 2026-02-13T02:06:48.179 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:06:48.179 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:06:48.179 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:06:48.179 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:06:48.179 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:06:48.180 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:06:48.180 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:06:48.180 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:06:48.180 INFO:tasks.workunit.client.0.trial112.stderr:185971: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:06:48.196 INFO:tasks.workunit.client.0.trial112.stderr:185971: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.svery_ops 2026-02-13T02:06:48.196 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2026-02-13T02:06:48.196 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2026-02-13T02:06:48.196 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:06:48.197 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:06:48.197 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:06:48.197 INFO:tasks.workunit.client.0.trial112.stderr:185969: 2026-02-13T02:06:46.704+0000 7f508045ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:06:48.197 INFO:tasks.workunit.client.0.trial112.stderr:185969: 2026-02-13T02:06:46.704+0000 7f508045ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:06:48.197 INFO:tasks.workunit.client.0.trial112.stderr:185969: 2026-02-13T02:06:46.704+0000 7f508045ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:06:48.197 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2026-02-13T02:06:48.197 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:06:48.197 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-13T02:06:48.197 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:06:48.197 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:06:48.197 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:06:48.197 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:06:48.198 INFO:tasks.workunit.client.0.trial112.stderr:185969: 0 2026-02-13T02:06:48.198 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:06:48.198 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-13T02:06:48.198 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:06:48.198 INFO:tasks.workunit.client.0.trial112.stderr:185969: 2026-02-13T02:06:47.184+0000 7f508045ea40 -1 Falling back to public interface 2026-02-13T02:06:48.198 INFO:tasks.workunit.client.0.trial112.stderr:185969: 2026-02-13T02:06:47.896+0000 7f508045ea40 -1 osd.5 91 log_to_monitors true 2026-02-13T02:06:48.198 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:06:48.198 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:06:48.198 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:06:48.198 INFO:tasks.workunit.client.0.trial112.stderr:185969: 1 2026-02-13T02:06:48.198 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:06:48.198 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-13T02:06:48.199 INFO:tasks.workunit.client.0.trial112.stderr:185969: osd.5 up in weight 1 up_from 95 up_thru 95 down_at 92 last_clean_interval [89,91) [v2:127.0.0.1:6842/2800037096,v1:127.0.0.1:6843/2800037096] [v2:127.0.0.1:6844/2800037096,v1:127.0.0.1:6845/2800037096] exists,up 670b1f5b-3687-4794-8a8a-767a3a2770e7 2026-02-13T02:06:48.199 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:06:48.199 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:06:48.199 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:06:48.199 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:06:48.199 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:06:48.199 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:06:48.199 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:06:48.199 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:06:48.199 INFO:tasks.workunit.client.0.trial112.stderr:185969: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:06:48.747 INFO:tasks.workunit.client.0.trial112.stderr:185969: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.h:1593: get_timeout_delays: echo true 2026-02-13T02:06:48.747 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:06:48.747 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:06:48.747 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:06:48.747 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:06:48.748 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:06:48.748 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:06:48.748 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:06:48.748 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:06:48.748 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:06:48.748 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:06:48.748 INFO:tasks.workunit.client.0.trial112.stderr:185971: 1 2026-02-13T02:06:48.748 INFO:tasks.workunit.client.0.trial112.stderr:185971: 2 2026-02-13T02:06:48.748 INFO:tasks.workunit.client.0.trial112.stderr:185971: 3 2026-02-13T02:06:48.748 INFO:tasks.workunit.client.0.trial112.stderr:185971: 4 2026-02-13T02:06:48.749 INFO:tasks.workunit.client.0.trial112.stderr:185971: 5 2026-02-13T02:06:48.749 INFO:tasks.workunit.client.0.trial112.stderr:185971: 6 2026-02-13T02:06:48.749 INFO:tasks.workunit.client.0.trial112.stderr:185971: 7 2026-02-13T02:06:48.749 INFO:tasks.workunit.client.0.trial112.stderr:185971: 8 2026-02-13T02:06:48.749 INFO:tasks.workunit.client.0.trial112.stderr:185971: 9' 2026-02-13T02:06:48.749 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:06:48.749 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:48.749 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:06:48.749 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836509 2026-02-13T02:06:48.749 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836509 2026-02-13T02:06:48.749 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509' 2026-02-13T02:06:48.749 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:48.750 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:06:48.750 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705693 2026-02-13T02:06:48.750 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705693 2026-02-13T02:06:48.750 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-38654705693' 2026-02-13T02:06:48.750 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:48.750 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:06:48.750 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574876 2026-02-13T02:06:48.750 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574876 2026-02-13T02:06:48.750 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-38654705693 2-55834574876' 2026-02-13T02:06:48.750 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:48.750 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:06:48.750 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=283467841554 2026-02-13T02:06:48.750 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 283467841554 2026-02-13T02:06:48.751 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-38654705693 2-55834574876 3-283467841554' 2026-02-13T02:06:48.751 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:48.769 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperssh:1593: get_timeout_delays: echo true 2026-02-13T02:06:48.769 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:06:48.769 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:06:48.769 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:06:48.769 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:06:48.770 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:06:48.770 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:06:48.770 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:06:48.770 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:06:48.770 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:06:48.770 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:06:48.770 INFO:tasks.workunit.client.0.trial112.stderr:185969: 1 2026-02-13T02:06:48.770 INFO:tasks.workunit.client.0.trial112.stderr:185969: 2 2026-02-13T02:06:48.770 INFO:tasks.workunit.client.0.trial112.stderr:185969: 3 2026-02-13T02:06:48.770 INFO:tasks.workunit.client.0.trial112.stderr:185969: 4 2026-02-13T02:06:48.770 INFO:tasks.workunit.client.0.trial112.stderr:185969: 5 2026-02-13T02:06:48.771 INFO:tasks.workunit.client.0.trial112.stderr:185969: 6 2026-02-13T02:06:48.771 INFO:tasks.workunit.client.0.trial112.stderr:185969: 7 2026-02-13T02:06:48.771 INFO:tasks.workunit.client.0.trial112.stderr:185969: 8 2026-02-13T02:06:48.771 INFO:tasks.workunit.client.0.trial112.stderr:185969: 9' 2026-02-13T02:06:48.771 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:06:48.771 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:48.771 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:06:48.771 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836510 2026-02-13T02:06:48.771 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836510 2026-02-13T02:06:48.771 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510' 2026-02-13T02:06:48.771 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:48.771 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:06:48.771 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705694 2026-02-13T02:06:48.772 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705694 2026-02-13T02:06:48.772 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-38654705694' 2026-02-13T02:06:48.772 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:48.772 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:06:48.772 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574877 2026-02-13T02:06:48.772 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574877 2026-02-13T02:06:48.772 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-38654705694 2-55834574877' 2026-02-13T02:06:48.772 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:48.772 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:06:48.772 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=283467841555 2026-02-13T02:06:48.772 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 283467841555 2026-02-13T02:06:48.772 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-38654705694 2-55834574877 3-283467841555' 2026-02-13T02:06:48.772 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:49.203 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-13T02:06:49.203 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280538 2026-02-13T02:06:49.203 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280538 2026-02-13T02:06:49.203 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-38654705693 2-55834574876 3-283467841554 4-94489280538' 2026-02-13T02:06:49.203 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:49.203 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-13T02:06:49.203 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=408021893122 2026-02-13T02:06:49.203 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 408021893122 2026-02-13T02:06:49.203 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-38654705693 2-55834574876 3-283467841554 4-94489280538 5-408021893122' 2026-02-13T02:06:49.204 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:49.204 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-13T02:06:49.204 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953498 2026-02-13T02:06:49.204 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953498 2026-02-13T02:06:49.204 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-38654705693 2-55834574876 3-283467841554 4-94489280538 5-408021893122 6-137438953498' 2026-02-13T02:06:49.204 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:49.204 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-13T02:06:49.204 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789976 2026-02-13T02:06:49.204 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789976 2026-02-13T02:06:49.204 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-38654705693 2-55834574876 3-283467841554 4-94489280538 5-408021893122 6-137438953498 7-158913789976' 2026-02-13T02:06:49.204 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:49.205 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-13T02:06:49.205 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626455 2026-02-13T02:06:49.205 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626455 2026-02-13T02:06:49.205 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-38654705693 2-55834574876 3-283467841554 4-94489280538 5-408021893122 6-137438953498 7-158913789976 8-180388626455' 2026-02-13T02:06:49.205 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:49.205 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-13T02:06:49.205 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=408021893122 2026-02-13T02:06:49.205 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 408021893122 2026-02-13T02:06:49.205 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-38654705693 2-55834574876 3-283467841554 4-94489280538 5-408021893122 6-137438953498 7-158913789976 8-180388626455 9-408021893122' 2026-02-13T02:06:49.205 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:49.206 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/us.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-13T02:06:49.206 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280539 2026-02-13T02:06:49.206 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280539 2026-02-13T02:06:49.206 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-38654705694 2-55834574877 3-283467841555 4-94489280539' 2026-02-13T02:06:49.206 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:49.206 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-13T02:06:49.207 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=408021893123 2026-02-13T02:06:49.207 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 408021893123 2026-02-13T02:06:49.207 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-38654705694 2-55834574877 3-283467841555 4-94489280539 5-408021893123' 2026-02-13T02:06:49.207 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:49.207 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-13T02:06:49.207 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953497 2026-02-13T02:06:49.207 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953497 2026-02-13T02:06:49.207 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-38654705694 2-55834574877 3-283467841555 4-94489280539 5-408021893123 6-137438953497' 2026-02-13T02:06:49.207 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:49.207 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-13T02:06:49.207 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789977 2026-02-13T02:06:49.207 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789977 2026-02-13T02:06:49.207 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-38654705694 2-55834574877 3-283467841555 4-94489280539 5-408021893123 6-137438953497 7-158913789977' 2026-02-13T02:06:49.208 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:49.208 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-13T02:06:49.208 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626456 2026-02-13T02:06:49.208 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626456 2026-02-13T02:06:49.208 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-38654705694 2-55834574877 3-283467841555 4-94489280539 5-408021893123 6-137438953497 7-158913789977 8-180388626456' 2026-02-13T02:06:49.208 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:49.208 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-13T02:06:49.208 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=408021893123 2026-02-13T02:06:49.208 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 408021893123 2026-02-13T02:06:49.208 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-38654705694 2-55834574877 3-283467841555 4-94489280539 5-408021893123 6-137438953497 7-158913789977 8-180388626456 9-408021893123' 2026-02-13T02:06:49.208 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:49.707 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836510 2026-02-13T02:06:49.707 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:49.707 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:06:49.708 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836510 2026-02-13T02:06:49.708 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:49.708 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836510 2026-02-13T02:06:49.708 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2026-02-13T02:06:49.708 INFO:tasks.workunit.client.0.trial112.stderr:185969: waiting osd.0 seq 21474836510 2026-02-13T02:06:49.708 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:06:49.708 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836510 -lt 21474836510 2026-02-13T02:06:49.708 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:49.708 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705694 2026-02-13T02:06:49.708 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:49.708 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:06:49.708 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705694 2026-02-13T02:06:49.708 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:49.708 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705694 2026-02-13T02:06:49.708 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705694' 2026-02-13T02:06:49.708 INFO:tasks.workunit.client.0.trial112.stderr:185969: waiting osd.1 seq 38654705694 2026-02-13T02:06:49.708 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:06:49.708 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705694 -lt 38654705694 2026-02-13T02:06:49.709 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:49.709 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574877 2026-02-13T02:06:49.709 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:49.709 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:06:49.709 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574877 2026-02-13T02:06:49.709 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:49.709 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574877 2026-02-13T02:06:49.709 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574877' 2026-02-13T02:06:49.709 INFO:tasks.workunit.client.0.trial112.stderr:185969: waiting osd.2 seq 55834574877 2026-02-13T02:06:49.709 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:06:49.709 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574877 -lt 55834574877 2026-02-13T02:06:49.709 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:49.709 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-283467841555 2026-02-13T02:06:49.709 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:49.709 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:06:49.724 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-283467buntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836509 2026-02-13T02:06:49.724 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:49.724 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:06:49.724 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836509 2026-02-13T02:06:49.724 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:49.724 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836509 2026-02-13T02:06:49.724 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2026-02-13T02:06:49.724 INFO:tasks.workunit.client.0.trial112.stderr:185971: waiting osd.0 seq 21474836509 2026-02-13T02:06:49.724 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:06:49.724 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836510 -lt 21474836509 2026-02-13T02:06:49.724 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:49.724 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705693 2026-02-13T02:06:49.724 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:49.724 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:06:49.724 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705693 2026-02-13T02:06:49.724 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:49.724 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705693 2026-02-13T02:06:49.724 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705693' 2026-02-13T02:06:49.724 INFO:tasks.workunit.client.0.trial112.stderr:185971: waiting osd.1 seq 38654705693 2026-02-13T02:06:49.724 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:06:49.724 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705694 -lt 38654705693 2026-02-13T02:06:49.724 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:49.724 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574876 2026-02-13T02:06:49.724 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:49.724 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:06:49.724 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574876 2026-02-13T02:06:49.724 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:49.724 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574876 2026-02-13T02:06:49.724 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574876' 2026-02-13T02:06:49.724 INFO:tasks.workunit.client.0.trial112.stderr:185971: waiting osd.2 seq 55834574876 2026-02-13T02:06:49.724 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:06:49.724 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574877 -lt 55834574876 2026-02-13T02:06:49.724 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:49.724 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-283467841554 2026-02-13T02:06:49.725 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:49.725 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:06:50.203 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-2834678841555 2026-02-13T02:06:50.203 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:50.203 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=283467841555 2026-02-13T02:06:50.203 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 283467841555' 2026-02-13T02:06:50.203 INFO:tasks.workunit.client.0.trial112.stderr:185969: waiting osd.3 seq 283467841555 2026-02-13T02:06:50.203 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:06:50.203 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 283467841555 -lt 283467841555 2026-02-13T02:06:50.203 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:50.203 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-94489280539 2026-02-13T02:06:50.204 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:50.204 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-13T02:06:50.204 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-94489280539 2026-02-13T02:06:50.204 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:50.204 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280539 2026-02-13T02:06:50.204 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 94489280539' 2026-02-13T02:06:50.204 INFO:tasks.workunit.client.0.trial112.stderr:185969: waiting osd.4 seq 94489280539 2026-02-13T02:06:50.204 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-13T02:06:50.204 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280539 -lt 94489280539 2026-02-13T02:06:50.204 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:50.204 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-408021893123 2026-02-13T02:06:50.204 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:50.205 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-13T02:06:50.205 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-408021893123 2026-02-13T02:06:50.205 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:50.205 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=408021893123 2026-02-13T02:06:50.205 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 408021893123' 2026-02-13T02:06:50.205 INFO:tasks.workunit.client.0.trial112.stderr:185969: waiting osd.5 seq 408021893123 2026-02-13T02:06:50.205 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-13T02:06:50.205 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 408021893123 -lt 408021893123 2026-02-13T02:06:50.205 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:50.205 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953497 2026-02-13T02:06:50.205 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:50.205 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-13T02:06:50.205 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953497 2026-02-13T02:06:50.206 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:50.206 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953497 2026-02-13T02:06:50.206 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953497' 2026-02-13T02:06:50.206 INFO:tasks.workunit.client.0.trial112.stderr:185969: waiting osd.6 seq 137438953497 2026-02-13T02:06:50.236 INFO:tasks.workunit.client.0.trial112.stderr:185969: /41554 2026-02-13T02:06:50.236 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:50.236 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=283467841554 2026-02-13T02:06:50.237 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 283467841554' 2026-02-13T02:06:50.237 INFO:tasks.workunit.client.0.trial112.stderr:185971: waiting osd.3 seq 283467841554 2026-02-13T02:06:50.237 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:06:50.237 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 283467841555 -lt 283467841554 2026-02-13T02:06:50.237 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:50.237 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-94489280538 2026-02-13T02:06:50.238 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:50.238 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-13T02:06:50.238 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-94489280538 2026-02-13T02:06:50.238 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:50.238 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280538 2026-02-13T02:06:50.239 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 94489280538' 2026-02-13T02:06:50.239 INFO:tasks.workunit.client.0.trial112.stderr:185971: waiting osd.4 seq 94489280538 2026-02-13T02:06:50.239 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-13T02:06:50.239 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280539 -lt 94489280538 2026-02-13T02:06:50.239 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:50.239 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-408021893122 2026-02-13T02:06:50.240 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:50.240 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-13T02:06:50.240 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-408021893122 2026-02-13T02:06:50.240 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:50.240 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=408021893122 2026-02-13T02:06:50.240 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 408021893122' 2026-02-13T02:06:50.241 INFO:tasks.workunit.client.0.trial112.stderr:185971: waiting osd.5 seq 408021893122 2026-02-13T02:06:50.241 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-13T02:06:50.241 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 408021893123 -lt 408021893122 2026-02-13T02:06:50.241 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:50.241 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953498 2026-02-13T02:06:50.241 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:50.242 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-13T02:06:50.242 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953498 2026-02-13T02:06:50.242 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:50.242 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953498 2026-02-13T02:06:50.242 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953498' 2026-02-13T02:06:50.242 INFO:tasks.workunit.client.0.trial112.stderr:185971: waiting osd.6 seq 137438953498 2026-02-13T02:06:50.885 INFO:tasks.workunit.client.0.trial112.stderr:185971: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-13T02:06:50.885 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953498 -lt 137438953497 2026-02-13T02:06:50.885 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:50.885 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789977 2026-02-13T02:06:50.885 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:50.885 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-13T02:06:50.885 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789977 2026-02-13T02:06:50.885 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:50.885 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789977 2026-02-13T02:06:50.885 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789977' 2026-02-13T02:06:50.885 INFO:tasks.workunit.client.0.trial112.stderr:185969: waiting osd.7 seq 158913789977 2026-02-13T02:06:50.885 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-13T02:06:50.885 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789977 -lt 158913789977 2026-02-13T02:06:50.885 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:50.885 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626456 2026-02-13T02:06:50.885 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:50.886 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-13T02:06:50.886 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626456 2026-02-13T02:06:50.886 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:50.886 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626456 2026-02-13T02:06:50.886 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626456' 2026-02-13T02:06:50.886 INFO:tasks.workunit.client.0.trial112.stderr:185969: waiting osd.8 seq 180388626456 2026-02-13T02:06:50.886 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-13T02:06:50.886 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626456 -lt 180388626456 2026-02-13T02:06:50.886 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:50.886 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-408021893123 2026-02-13T02:06:50.886 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:50.886 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-13T02:06:50.886 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-408021893123 2026-02-13T02:06:50.886 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:50.886 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=408021893123 2026-02-13T02:06:50.886 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 408021893123' 2026-02-13T02:06:50.886 INFO:tasks.workunit.client.0.trial112.stderr:185969: waiting osd.9 seq 408021893123 2026-02-13T02:06:50.886 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-13T02:06:50.886 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 408021893123 -lt 408021893123 2026-02-13T02:06:50.886 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:06:50.906 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/stanhome/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-13T02:06:50.907 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953498 -lt 137438953498 2026-02-13T02:06:50.907 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:50.907 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789976 2026-02-13T02:06:50.907 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:50.907 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-13T02:06:50.907 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789976 2026-02-13T02:06:50.907 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:50.907 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789976 2026-02-13T02:06:50.907 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789976' 2026-02-13T02:06:50.907 INFO:tasks.workunit.client.0.trial112.stderr:185971: waiting osd.7 seq 158913789976 2026-02-13T02:06:50.907 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-13T02:06:50.907 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789977 -lt 158913789976 2026-02-13T02:06:50.907 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:50.907 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626455 2026-02-13T02:06:50.907 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:50.907 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-13T02:06:50.907 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626455 2026-02-13T02:06:50.907 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:50.907 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626455 2026-02-13T02:06:50.907 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626455' 2026-02-13T02:06:50.907 INFO:tasks.workunit.client.0.trial112.stderr:185971: waiting osd.8 seq 180388626455 2026-02-13T02:06:50.908 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-13T02:06:50.908 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626456 -lt 180388626455 2026-02-13T02:06:50.908 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:50.908 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-408021893122 2026-02-13T02:06:50.908 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:50.908 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-13T02:06:50.908 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-408021893122 2026-02-13T02:06:50.908 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:50.908 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=408021893122 2026-02-13T02:06:50.908 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 408021893122' 2026-02-13T02:06:50.908 INFO:tasks.workunit.client.0.trial112.stderr:185971: waiting osd.9 seq 408021893122 2026-02-13T02:06:50.908 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-13T02:06:50.908 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 408021893123 -lt 408021893122 2026-02-13T02:06:50.908 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:06:51.433 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standdalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:06:51.433 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:06:51.433 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:06:51.433 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:06:51.433 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:06:51.433 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:06:51.433 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:06:51.433 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:06:51.434 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:06:51.434 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:06:51.434 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:06:51.434 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:06:51.434 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:06:51.434 INFO:tasks.workunit.client.0.trial112.stderr:185969: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:06:51.434 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:06:51.434 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:06:51.434 INFO:tasks.workunit.client.0.trial112.stderr:185969: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:06:51.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-13T02:06:51.435 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-13T02:06:51.435 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 185968 2026-02-13T02:06:51.468 INFO:tasks.workunit.client.0.trial112.stderr:alone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:06:51.468 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:06:51.469 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:06:51.469 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:06:51.469 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:06:51.469 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:06:51.469 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:06:51.469 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:06:51.469 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:06:51.469 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:06:51.469 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:06:51.469 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:06:51.469 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:06:51.470 INFO:tasks.workunit.client.0.trial112.stderr:185971: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:06:51.470 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:06:51.470 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:06:51.470 INFO:tasks.workunit.client.0.trial112.stderr:185971: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:06:51.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-13T02:06:51.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-13T02:06:51.471 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-13T02:06:51.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-13T02:06:51.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:272: corrupt_and_repair_two: return_code=0 2026-02-13T02:06:51.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-13T02:06:51.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:275: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-13T02:06:51.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:276: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-13T02:06:51.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:323: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2026-02-13T02:06:51.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:245: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2026-02-13T02:06:51.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:246: corrupt_and_repair_two: local poolname=ecpool 2026-02-13T02:06:51.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:247: corrupt_and_repair_two: local first=3 2026-02-13T02:06:51.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_two: local second=5 2026-02-13T02:06:51.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_two: pids= 2026-02-13T02:06:51.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-13T02:06:51.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-13T02:06:51.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-13T02:06:51.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 188793"' 2026-02-13T02:06:51.492 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 188793' 2026-02-13T02:06:51.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2026-02-13T02:06:51.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-13T02:06:51.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-13T02:06:51.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 188795"' 2026-02-13T02:06:51.493 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 188795' 2026-02-13T02:06:51.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-13T02:06:51.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_two: wait_background pids 2026-02-13T02:06:51.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 188793 188795' 2026-02-13T02:06:51.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-13T02:06:51.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-13T02:06:51.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 188793 2026-02-13T02:06:51.494 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/188796: /' 2026-02-13T02:06:51.494 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2026-02-13T02:06:51.494 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/188798: /' 2026-02-13T02:06:52.102 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:06:52.102 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:06:52.102 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2026-02-13T02:06:52.102 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:06:52.103 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2026-02-13T02:06:52.103 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:06:52.103 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:06:52.103 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2026-02-13T02:06:52.103 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:06:52.103 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2026-02-13T02:06:52.103 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:06:52.103 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:06:52.103 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:06:52.103 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:06:52.103 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:06:52.104 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:06:52.104 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2026-02-13T02:06:52.104 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:06:52.104 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:06:52.104 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2026-02-13T02:06:52.104 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:06:52.104 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2026-02-13T02:06:52.104 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2026-02-13T02:06:52.104 INFO:tasks.workunit.client.0.trial112.stderr:188798: remove 1#2:eb822e21:::SOMETHING:head# 2026-02-13T02:06:52.104 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2026-02-13T02:06:52.104 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:06:52.104 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:06:52.105 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2026-02-13T02:06:52.105 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:06:52.105 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2026-02-13T02:06:52.105 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:06:52.105 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:06:52.106 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:06:52.106 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:06:52.106 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2026-02-13T02:06:52.106 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2026-02-13T02:06:52.106 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:06:52.107 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:06:52.107 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:06:52.107 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:06:52.107 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:06:52.107 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:06:52.107 INFO:tasks.workunit.client.0.trial112.stderr:188798: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:06:52.107 INFO:tasks.workunit.client.0.trial112.stderr:188798: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:06:52.107 INFO:tasks.workunit.client.0.trial112.stderr:188798: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:06:52.107 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:06:52.107 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:06:52.107 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:06:52.107 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:06:52.108 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:06:52.108 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:06:52.108 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:06:52.108 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:06:52.108 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:06:52.108 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:06:52.108 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:06:52.108 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2026-02-13T02:06:52.108 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2026-02-13T02:06:52.108 INFO:tasks.workunit.client.0.trial112.stderr:188798: start osd.5 2026-02-13T02:06:52.108 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:06:52.142 INFO:tasks.workunit.client.0.trial112.stderr:1188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:06:52.142 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:06:52.142 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-13T02:06:52.142 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:06:52.142 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2026-02-13T02:06:52.143 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:06:52.143 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:06:52.143 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-13T02:06:52.143 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:06:52.143 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-13T02:06:52.143 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:06:52.143 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:06:52.143 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:06:52.143 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:06:52.143 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:06:52.143 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:06:52.143 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2026-02-13T02:06:52.144 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:06:52.144 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:06:52.144 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-13T02:06:52.144 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:06:52.144 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:06:52.144 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2026-02-13T02:06:52.144 INFO:tasks.workunit.client.0.trial112.stderr:188796: remove 0#2:eb822e21:::SOMETHING:head# 2026-02-13T02:06:52.144 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-13T02:06:52.144 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:06:52.144 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:06:52.144 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-13T02:06:52.145 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:06:52.145 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:06:52.145 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:06:52.145 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:06:52.145 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:06:52.145 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:06:52.146 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-13T02:06:52.146 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-13T02:06:52.146 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:06:52.146 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:06:52.146 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:06:52.146 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:06:52.146 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:06:52.146 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:06:52.146 INFO:tasks.workunit.client.0.trial112.stderr:188796: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:06:52.146 INFO:tasks.workunit.client.0.trial112.stderr:188796: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:06:52.146 INFO:tasks.workunit.client.0.trial112.stderr:188796: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:06:52.147 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:06:52.147 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:06:52.147 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:06:52.147 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:06:52.147 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:06:52.147 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:06:52.147 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:06:52.147 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:06:52.147 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:06:52.147 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:06:52.147 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:06:52.147 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-13T02:06:52.148 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-13T02:06:52.148 INFO:tasks.workunit.client.0.trial112.stderr:188796: start osd.3 2026-02-13T02:06:52.148 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:06:53.591 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2026-02-13T02:06:53.591 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2026-02-13T02:06:53.591 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:06:53.591 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:06:53.591 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:06:53.591 INFO:tasks.workunit.client.0.trial112.stderr:188798: 2026-02-13T02:06:52.116+0000 7f824f561a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:06:53.592 INFO:tasks.workunit.client.0.trial112.stderr:188798: 2026-02-13T02:06:52.116+0000 7f824f561a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:06:53.592 INFO:tasks.workunit.client.0.trial112.stderr:188798: 2026-02-13T02:06:52.116+0000 7f824f561a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:06:53.592 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2026-02-13T02:06:53.592 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:06:53.592 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-13T02:06:53.592 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:06:53.592 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:06:53.592 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:06:53.592 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:06:53.592 INFO:tasks.workunit.client.0.trial112.stderr:188798: 0 2026-02-13T02:06:53.592 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:06:53.593 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-13T02:06:53.593 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:06:53.593 INFO:tasks.workunit.client.0.trial112.stderr:188798: 2026-02-13T02:06:52.600+0000 7f824f561a40 -1 Falling back to public interface 2026-02-13T02:06:53.593 INFO:tasks.workunit.client.0.trial112.stderr:188798: 2026-02-13T02:06:53.220+0000 7f824f561a40 -1 osd.5 96 log_to_monitors true 2026-02-13T02:06:53.593 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:06:53.593 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:06:53.593 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:06:53.593 INFO:tasks.workunit.client.0.trial112.stderr:188798: 1 2026-02-13T02:06:53.593 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:06:53.593 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-13T02:06:53.593 INFO:tasks.workunit.client.0.trial112.stderr:188798: osd.5 up in weight 1 up_from 100 up_thru 100 down_at 97 last_clean_interval [95,96) [v2:127.0.0.1:6826/2901136111,v1:127.0.0.1:6827/2901136111] [v2:127.0.0.1:6828/2901136111,v1:127.0.0.1:6829/2901136111] exists,up 670b1f5b-3687-4794-8a8a-767a3a2770e7 2026-02-13T02:06:53.593 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:06:53.593 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:06:53.594 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:06:53.594 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:06:53.594 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:06:53.594 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:06:53.594 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:06:53.594 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:06:53.594 INFO:tasks.workunit.client.0.trial112.stderr:188798: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:06:53.639 INFO:tasks.workunit.client.0.trial112.stderr:188798: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593:88796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-13T02:06:53.639 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-13T02:06:53.639 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:06:53.639 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:06:53.639 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:06:53.639 INFO:tasks.workunit.client.0.trial112.stderr:188796: 2026-02-13T02:06:52.148+0000 7f6427e22a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:06:53.639 INFO:tasks.workunit.client.0.trial112.stderr:188796: 2026-02-13T02:06:52.148+0000 7f6427e22a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:06:53.639 INFO:tasks.workunit.client.0.trial112.stderr:188796: 2026-02-13T02:06:52.152+0000 7f6427e22a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:06:53.639 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-13T02:06:53.640 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:06:53.640 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-13T02:06:53.640 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:06:53.640 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:06:53.640 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:06:53.640 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:06:53.640 INFO:tasks.workunit.client.0.trial112.stderr:188796: 0 2026-02-13T02:06:53.640 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:06:53.640 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:06:53.640 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:06:53.640 INFO:tasks.workunit.client.0.trial112.stderr:188796: 2026-02-13T02:06:52.648+0000 7f6427e22a40 -1 Falling back to public interface 2026-02-13T02:06:53.641 INFO:tasks.workunit.client.0.trial112.stderr:188796: 2026-02-13T02:06:53.316+0000 7f6427e22a40 -1 osd.3 96 log_to_monitors true 2026-02-13T02:06:53.641 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:06:53.641 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:06:53.641 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:06:53.641 INFO:tasks.workunit.client.0.trial112.stderr:188796: 1 2026-02-13T02:06:53.641 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:06:53.641 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:06:53.641 INFO:tasks.workunit.client.0.trial112.stderr:188796: osd.3 up in weight 1 up_from 101 up_thru 101 down_at 97 last_clean_interval [66,96) [v2:127.0.0.1:6842/578237917,v1:127.0.0.1:6843/578237917] [v2:127.0.0.1:6844/578237917,v1:127.0.0.1:6845/578237917] exists,up ad30d011-cd62-4310-8456-c2d78faa8a93 2026-02-13T02:06:53.641 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:06:53.641 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:06:53.641 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:06:53.641 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:06:53.642 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:06:53.642 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:06:53.642 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:06:53.642 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:06:53.642 INFO:tasks.workunit.client.0.trial112.stderr:188796: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:06:54.143 INFO:tasks.workunit.client.0.trial112.stderr:188796: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get get_timeout_delays: echo true 2026-02-13T02:06:54.143 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:06:54.143 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:06:54.143 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:06:54.143 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:06:54.143 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:06:54.143 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:06:54.143 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:06:54.144 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:06:54.144 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:06:54.144 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:06:54.144 INFO:tasks.workunit.client.0.trial112.stderr:188798: 1 2026-02-13T02:06:54.144 INFO:tasks.workunit.client.0.trial112.stderr:188798: 2 2026-02-13T02:06:54.144 INFO:tasks.workunit.client.0.trial112.stderr:188798: 3 2026-02-13T02:06:54.144 INFO:tasks.workunit.client.0.trial112.stderr:188798: 4 2026-02-13T02:06:54.144 INFO:tasks.workunit.client.0.trial112.stderr:188798: 5 2026-02-13T02:06:54.144 INFO:tasks.workunit.client.0.trial112.stderr:188798: 6 2026-02-13T02:06:54.144 INFO:tasks.workunit.client.0.trial112.stderr:188798: 7 2026-02-13T02:06:54.144 INFO:tasks.workunit.client.0.trial112.stderr:188798: 8 2026-02-13T02:06:54.144 INFO:tasks.workunit.client.0.trial112.stderr:188798: 9' 2026-02-13T02:06:54.145 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:06:54.145 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:54.145 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:06:54.145 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836512 2026-02-13T02:06:54.145 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836512 2026-02-13T02:06:54.145 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512' 2026-02-13T02:06:54.145 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:54.145 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:06:54.145 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705696 2026-02-13T02:06:54.145 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705696 2026-02-13T02:06:54.145 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512 1-38654705696' 2026-02-13T02:06:54.145 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:54.146 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:06:54.146 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574879 2026-02-13T02:06:54.146 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574879 2026-02-13T02:06:54.146 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512 1-38654705696 2-55834574879' 2026-02-13T02:06:54.146 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:54.146 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:06:54.146 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=433791696898 2026-02-13T02:06:54.146 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 433791696898 2026-02-13T02:06:54.146 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512 1-38654705696 2-55834574879 3-433791696898' 2026-02-13T02:06:54.146 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:54.198 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:225_timeout_delays: echo true 2026-02-13T02:06:54.199 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:06:54.199 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:06:54.199 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:06:54.199 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:06:54.199 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:06:54.199 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:06:54.199 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:06:54.199 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:06:54.199 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:06:54.199 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:06:54.200 INFO:tasks.workunit.client.0.trial112.stderr:188796: 1 2026-02-13T02:06:54.200 INFO:tasks.workunit.client.0.trial112.stderr:188796: 2 2026-02-13T02:06:54.200 INFO:tasks.workunit.client.0.trial112.stderr:188796: 3 2026-02-13T02:06:54.200 INFO:tasks.workunit.client.0.trial112.stderr:188796: 4 2026-02-13T02:06:54.200 INFO:tasks.workunit.client.0.trial112.stderr:188796: 5 2026-02-13T02:06:54.200 INFO:tasks.workunit.client.0.trial112.stderr:188796: 6 2026-02-13T02:06:54.200 INFO:tasks.workunit.client.0.trial112.stderr:188796: 7 2026-02-13T02:06:54.200 INFO:tasks.workunit.client.0.trial112.stderr:188796: 8 2026-02-13T02:06:54.200 INFO:tasks.workunit.client.0.trial112.stderr:188796: 9' 2026-02-13T02:06:54.200 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:06:54.200 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:54.201 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:06:54.201 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836513 2026-02-13T02:06:54.201 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836513 2026-02-13T02:06:54.201 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513' 2026-02-13T02:06:54.201 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:54.201 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:06:54.201 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705697 2026-02-13T02:06:54.201 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705697 2026-02-13T02:06:54.201 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-38654705697' 2026-02-13T02:06:54.201 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:54.201 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:06:54.202 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574880 2026-02-13T02:06:54.202 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574880 2026-02-13T02:06:54.202 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-38654705697 2-55834574880' 2026-02-13T02:06:54.202 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:54.202 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:06:54.202 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=433791696899 2026-02-13T02:06:54.202 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 433791696899 2026-02-13T02:06:54.202 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-38654705697 2-55834574880 3-433791696899' 2026-02-13T02:06:54.202 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:54.578 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: f8: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-13T02:06:54.578 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280541 2026-02-13T02:06:54.578 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280541 2026-02-13T02:06:54.578 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512 1-38654705696 2-55834574879 3-433791696898 4-94489280541' 2026-02-13T02:06:54.579 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:54.579 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-13T02:06:54.579 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=429496729602 2026-02-13T02:06:54.579 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 429496729602 2026-02-13T02:06:54.579 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512 1-38654705696 2-55834574879 3-433791696898 4-94489280541 5-429496729602' 2026-02-13T02:06:54.579 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:54.579 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-13T02:06:54.579 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953500 2026-02-13T02:06:54.579 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953500 2026-02-13T02:06:54.579 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512 1-38654705696 2-55834574879 3-433791696898 4-94489280541 5-429496729602 6-137438953500' 2026-02-13T02:06:54.579 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:54.580 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-13T02:06:54.580 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789979 2026-02-13T02:06:54.580 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789979 2026-02-13T02:06:54.580 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512 1-38654705696 2-55834574879 3-433791696898 4-94489280541 5-429496729602 6-137438953500 7-158913789979' 2026-02-13T02:06:54.580 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:54.580 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-13T02:06:54.580 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626458 2026-02-13T02:06:54.580 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626458 2026-02-13T02:06:54.580 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512 1-38654705696 2-55834574879 3-433791696898 4-94489280541 5-429496729602 6-137438953500 7-158913789979 8-180388626458' 2026-02-13T02:06:54.580 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:54.580 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-13T02:06:54.581 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=408021893125 2026-02-13T02:06:54.581 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 408021893125 2026-02-13T02:06:54.581 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512 1-38654705696 2-55834574879 3-433791696898 4-94489280541 5-429496729602 6-137438953500 7-158913789979 8-180388626458 9-408021893125' 2026-02-13T02:06:54.581 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:54.655 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/clush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-13T02:06:54.655 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280542 2026-02-13T02:06:54.655 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280542 2026-02-13T02:06:54.655 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-38654705697 2-55834574880 3-433791696899 4-94489280542' 2026-02-13T02:06:54.655 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:54.655 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-13T02:06:54.655 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=429496729603 2026-02-13T02:06:54.655 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 429496729603 2026-02-13T02:06:54.655 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-38654705697 2-55834574880 3-433791696899 4-94489280542 5-429496729603' 2026-02-13T02:06:54.656 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:54.656 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-13T02:06:54.656 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953501 2026-02-13T02:06:54.656 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953501 2026-02-13T02:06:54.656 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-38654705697 2-55834574880 3-433791696899 4-94489280542 5-429496729603 6-137438953501' 2026-02-13T02:06:54.656 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:54.656 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-13T02:06:54.656 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789980 2026-02-13T02:06:54.656 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789980 2026-02-13T02:06:54.656 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-38654705697 2-55834574880 3-433791696899 4-94489280542 5-429496729603 6-137438953501 7-158913789980' 2026-02-13T02:06:54.656 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:54.656 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-13T02:06:54.657 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626460 2026-02-13T02:06:54.657 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626460 2026-02-13T02:06:54.657 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-38654705697 2-55834574880 3-433791696899 4-94489280542 5-429496729603 6-137438953501 7-158913789980 8-180388626460' 2026-02-13T02:06:54.657 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:06:54.657 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-13T02:06:54.657 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=408021893126 2026-02-13T02:06:54.657 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 408021893126 2026-02-13T02:06:54.657 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-38654705697 2-55834574880 3-433791696899 4-94489280542 5-429496729603 6-137438953501 7-158913789980 8-180388626460 9-408021893126' 2026-02-13T02:06:54.657 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:56.249 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836512 2026-02-13T02:06:56.249 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:56.249 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:06:56.249 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836512 2026-02-13T02:06:56.249 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:56.250 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836512 2026-02-13T02:06:56.250 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2026-02-13T02:06:56.250 INFO:tasks.workunit.client.0.trial112.stderr:188798: waiting osd.0 seq 21474836512 2026-02-13T02:06:56.250 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:06:56.250 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836511 -lt 21474836512 2026-02-13T02:06:56.250 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:06:56.250 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:06:56.250 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:06:56.250 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836513 -lt 21474836512 2026-02-13T02:06:56.250 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:56.250 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705696 2026-02-13T02:06:56.250 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:56.251 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:06:56.251 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705696 2026-02-13T02:06:56.251 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:56.251 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705696 2026-02-13T02:06:56.251 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705696' 2026-02-13T02:06:56.251 INFO:tasks.workunit.client.0.trial112.stderr:188798: waiting osd.1 seq 38654705696 2026-02-13T02:06:56.251 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:06:56.251 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705697 -lt 38654705696 2026-02-13T02:06:56.251 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:56.251 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574879 2026-02-13T02:06:56.251 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:56.251 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:06:56.251 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574879 2026-02-13T02:06:56.252 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:56.252 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574879 2026-02-13T02:06:56.252 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574879' 2026-02-13T02:06:56.252 INFO:tasks.workunit.client.0.trial112.stderr:188798: waiting osd.2 seq 55834574879 2026-02-13T02:06:56.252 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:06:56.252 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574880 -lt 55834574879 2026-02-13T02:06:56.311 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836513 2026-02-13T02:06:56.311 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:56.312 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:06:56.312 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836513 2026-02-13T02:06:56.312 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:56.312 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836513 2026-02-13T02:06:56.312 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2026-02-13T02:06:56.312 INFO:tasks.workunit.client.0.trial112.stderr:188796: waiting osd.0 seq 21474836513 2026-02-13T02:06:56.312 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:06:56.312 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836511 -lt 21474836513 2026-02-13T02:06:56.312 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:06:56.312 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:06:56.312 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:06:56.312 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836513 -lt 21474836513 2026-02-13T02:06:56.313 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:56.313 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705697 2026-02-13T02:06:56.313 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:56.313 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:06:56.313 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705697 2026-02-13T02:06:56.313 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:56.313 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705697 2026-02-13T02:06:56.313 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705697' 2026-02-13T02:06:56.313 INFO:tasks.workunit.client.0.trial112.stderr:188796: waiting osd.1 seq 38654705697 2026-02-13T02:06:56.313 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:06:56.313 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705697 -lt 38654705697 2026-02-13T02:06:56.313 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:56.314 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574880 2026-02-13T02:06:56.314 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:56.314 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:06:56.314 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574880 2026-02-13T02:06:56.314 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:56.314 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574880 2026-02-13T02:06:56.314 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574880' 2026-02-13T02:06:56.314 INFO:tasks.workunit.client.0.trial112.stderr:188796: waiting osd.2 seq 55834574880 2026-02-13T02:06:56.314 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:06:56.314 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574880 -lt 55834574880 2026-02-13T02:06:56.769 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats:ats: for s in $seqs 2026-02-13T02:06:56.769 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-433791696898 2026-02-13T02:06:56.769 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:56.769 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:06:56.769 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-433791696898 2026-02-13T02:06:56.769 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:56.769 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=433791696898 2026-02-13T02:06:56.769 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 433791696898' 2026-02-13T02:06:56.769 INFO:tasks.workunit.client.0.trial112.stderr:188798: waiting osd.3 seq 433791696898 2026-02-13T02:06:56.770 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:06:56.770 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 433791696899 -lt 433791696898 2026-02-13T02:06:56.770 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:56.770 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-94489280541 2026-02-13T02:06:56.770 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:56.770 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-13T02:06:56.770 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-94489280541 2026-02-13T02:06:56.770 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:56.770 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280541 2026-02-13T02:06:56.770 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 94489280541' 2026-02-13T02:06:56.770 INFO:tasks.workunit.client.0.trial112.stderr:188798: waiting osd.4 seq 94489280541 2026-02-13T02:06:56.770 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-13T02:06:56.771 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280543 -lt 94489280541 2026-02-13T02:06:56.771 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:56.771 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-429496729602 2026-02-13T02:06:56.771 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:56.771 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-13T02:06:56.771 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-429496729602 2026-02-13T02:06:56.771 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:56.771 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=429496729602 2026-02-13T02:06:56.771 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 429496729602' 2026-02-13T02:06:56.771 INFO:tasks.workunit.client.0.trial112.stderr:188798: waiting osd.5 seq 429496729602 2026-02-13T02:06:56.771 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-13T02:06:56.771 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 429496729603 -lt 429496729602 2026-02-13T02:06:56.771 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:56.772 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953500 2026-02-13T02:06:56.772 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:56.772 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-13T02:06:56.800 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ce for s in $seqs 2026-02-13T02:06:56.800 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-433791696899 2026-02-13T02:06:56.800 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:56.800 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:06:56.800 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-433791696899 2026-02-13T02:06:56.800 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:56.800 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=433791696899 2026-02-13T02:06:56.800 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 433791696899' 2026-02-13T02:06:56.800 INFO:tasks.workunit.client.0.trial112.stderr:188796: waiting osd.3 seq 433791696899 2026-02-13T02:06:56.800 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:06:56.801 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 433791696899 -lt 433791696899 2026-02-13T02:06:56.801 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:56.801 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-94489280542 2026-02-13T02:06:56.801 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:56.801 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-13T02:06:56.801 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-94489280542 2026-02-13T02:06:56.801 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:56.801 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280542 2026-02-13T02:06:56.801 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 94489280542' 2026-02-13T02:06:56.801 INFO:tasks.workunit.client.0.trial112.stderr:188796: waiting osd.4 seq 94489280542 2026-02-13T02:06:56.801 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-13T02:06:56.801 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280543 -lt 94489280542 2026-02-13T02:06:56.801 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:56.802 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-429496729603 2026-02-13T02:06:56.802 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:56.802 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-13T02:06:56.802 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-429496729603 2026-02-13T02:06:56.802 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:56.802 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=429496729603 2026-02-13T02:06:56.802 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 429496729603' 2026-02-13T02:06:56.802 INFO:tasks.workunit.client.0.trial112.stderr:188796: waiting osd.5 seq 429496729603 2026-02-13T02:06:56.802 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-13T02:06:56.802 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 429496729603 -lt 429496729603 2026-02-13T02:06:56.802 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:56.802 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953501 2026-02-13T02:06:56.802 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:56.803 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-13T02:06:57.288 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-hph-helpers.sh:2268: flush_pg_stats: echo 6-137438953500 2026-02-13T02:06:57.289 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:57.289 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953500 2026-02-13T02:06:57.289 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953500' 2026-02-13T02:06:57.289 INFO:tasks.workunit.client.0.trial112.stderr:188798: waiting osd.6 seq 137438953500 2026-02-13T02:06:57.289 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-13T02:06:57.289 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953501 -lt 137438953500 2026-02-13T02:06:57.289 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:57.289 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789979 2026-02-13T02:06:57.289 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:57.289 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-13T02:06:57.289 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789979 2026-02-13T02:06:57.290 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:57.290 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789979 2026-02-13T02:06:57.290 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789979' 2026-02-13T02:06:57.290 INFO:tasks.workunit.client.0.trial112.stderr:188798: waiting osd.7 seq 158913789979 2026-02-13T02:06:57.290 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-13T02:06:57.290 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789980 -lt 158913789979 2026-02-13T02:06:57.290 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:57.290 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626458 2026-02-13T02:06:57.290 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:57.290 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-13T02:06:57.290 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626458 2026-02-13T02:06:57.290 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:57.291 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626458 2026-02-13T02:06:57.291 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626458' 2026-02-13T02:06:57.291 INFO:tasks.workunit.client.0.trial112.stderr:188798: waiting osd.8 seq 180388626458 2026-02-13T02:06:57.291 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-13T02:06:57.291 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626460 -lt 180388626458 2026-02-13T02:06:57.291 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:57.291 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-408021893125 2026-02-13T02:06:57.291 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:57.291 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-13T02:06:57.291 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-408021893125 2026-02-13T02:06:57.291 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:57.291 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=408021893125 2026-02-13T02:06:57.301 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 40802elpers.sh:2268: flush_pg_stats: echo 6-137438953501 2026-02-13T02:06:57.301 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:57.301 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953501 2026-02-13T02:06:57.301 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953501' 2026-02-13T02:06:57.301 INFO:tasks.workunit.client.0.trial112.stderr:188796: waiting osd.6 seq 137438953501 2026-02-13T02:06:57.301 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-13T02:06:57.302 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953501 -lt 137438953501 2026-02-13T02:06:57.302 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:57.302 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789980 2026-02-13T02:06:57.302 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:57.302 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-13T02:06:57.302 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789980 2026-02-13T02:06:57.302 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:57.302 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789980 2026-02-13T02:06:57.302 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789980' 2026-02-13T02:06:57.302 INFO:tasks.workunit.client.0.trial112.stderr:188796: waiting osd.7 seq 158913789980 2026-02-13T02:06:57.302 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-13T02:06:57.302 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789980 -lt 158913789980 2026-02-13T02:06:57.302 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:57.303 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626460 2026-02-13T02:06:57.303 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:57.303 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-13T02:06:57.303 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626460 2026-02-13T02:06:57.303 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:57.303 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626460 2026-02-13T02:06:57.303 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626460' 2026-02-13T02:06:57.303 INFO:tasks.workunit.client.0.trial112.stderr:188796: waiting osd.8 seq 180388626460 2026-02-13T02:06:57.303 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-13T02:06:57.303 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626460 -lt 180388626460 2026-02-13T02:06:57.303 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:06:57.303 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-408021893126 2026-02-13T02:06:57.304 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:06:57.304 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-13T02:06:57.304 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-408021893126 2026-02-13T02:06:57.304 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:06:57.304 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=408021893126 2026-02-13T02:06:58.282 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 408021893126' 2026-02-13T02:06:58.282 INFO:tasks.workunit.client.0.trial112.stderr:188796: waiting osd.9 seq 408021893126 2026-02-13T02:06:58.282 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-13T02:06:58.283 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 408021893126 -lt 408021893126 2026-02-13T02:06:58.283 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:06:58.283 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:06:58.283 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:06:58.283 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:06:58.283 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:06:58.283 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:06:58.283 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:06:58.283 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:06:58.283 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:06:58.283 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:06:58.283 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:06:58.284 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-13T02:06:58.284 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:06:58.284 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:06:58.284 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:06:58.284 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-13T02:06:58.284 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' -1 2026-02-13T02:06:58.284 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-13T02:06:58.284 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=4 2026-02-13T02:06:58.284 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-13T02:06:58.284 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-13T02:06:58.284 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-13T02:06:58.284 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:06:58.285 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:06:58.285 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:06:58.285 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:06:58.285 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:06:58.285 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:06:58.285 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:06:58.290 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/1893125' 2026-02-13T02:06:58.290 INFO:tasks.workunit.client.0.trial112.stderr:188798: waiting osd.9 seq 408021893125 2026-02-13T02:06:58.290 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-13T02:06:58.290 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 408021893126 -lt 408021893125 2026-02-13T02:06:58.290 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:06:58.290 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:06:58.290 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:06:58.290 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:06:58.290 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:06:58.291 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:06:58.291 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:06:58.291 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:06:58.291 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:06:58.291 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:06:58.291 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:06:58.291 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-13T02:06:58.291 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:06:58.291 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:06:58.291 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:06:58.291 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-13T02:06:58.291 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' -1 2026-02-13T02:06:58.292 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-13T02:06:58.292 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=4 2026-02-13T02:06:58.292 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-13T02:06:58.292 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-13T02:06:58.292 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-13T02:06:58.292 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:06:58.292 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:06:58.292 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:06:58.292 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:06:58.292 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:06:58.292 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:06:58.292 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:06:59.141 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.clienqa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-13T02:06:59.141 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:06:59.141 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:06:59.141 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:06:59.141 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-13T02:06:59.141 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' 4 2026-02-13T02:06:59.141 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-13T02:06:59.141 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-13T02:06:59.141 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-13T02:06:59.142 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-13T02:06:59.142 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-13T02:06:59.142 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-13T02:06:59.142 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-13T02:06:59.142 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=5 2026-02-13T02:06:59.142 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 5 '!=' null 2026-02-13T02:06:59.142 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1668: wait_for_clean: loop=0 2026-02-13T02:06:59.142 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-13T02:06:59.142 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-13T02:06:59.142 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-13T02:06:59.142 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:06:59.142 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:06:59.143 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:06:59.143 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:06:59.143 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:06:59.143 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:06:59.143 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:06:59.143 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-13T02:06:59.143 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:06:59.143 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:06:59.143 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:06:59.143 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-13T02:06:59.149 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/t.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-13T02:06:59.149 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:06:59.149 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:06:59.149 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:06:59.149 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-13T02:06:59.149 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' 4 2026-02-13T02:06:59.149 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-13T02:06:59.150 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-13T02:06:59.150 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-13T02:06:59.150 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-13T02:06:59.150 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-13T02:06:59.150 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-13T02:06:59.150 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-13T02:06:59.150 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=5 2026-02-13T02:06:59.150 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 5 '!=' null 2026-02-13T02:06:59.150 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1668: wait_for_clean: loop=0 2026-02-13T02:06:59.150 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-13T02:06:59.150 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-13T02:06:59.150 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-13T02:06:59.151 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:06:59.151 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:06:59.151 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:06:59.151 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:06:59.151 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:06:59.151 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:06:59.151 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:06:59.151 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-13T02:06:59.151 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:06:59.151 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:06:59.151 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:06:59.152 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-13T02:06:59.914 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.clienqa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' 4 2026-02-13T02:06:59.914 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-13T02:06:59.914 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-13T02:06:59.914 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-13T02:06:59.914 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-13T02:06:59.914 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-13T02:06:59.914 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-13T02:06:59.915 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-13T02:06:59.915 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-13T02:06:59.915 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-13T02:06:59.915 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-13T02:06:59.915 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-13T02:06:59.915 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-13T02:06:59.915 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-13T02:06:59.915 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:06:59.915 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:06:59.915 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:06:59.915 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:06:59.915 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:06:59.916 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:06:59.916 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:06:59.916 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:06:59.916 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:06:59.916 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:06:59.916 INFO:tasks.workunit.client.0.trial112.stderr:188796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:06:59.916 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:06:59.916 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:06:59.916 INFO:tasks.workunit.client.0.trial112.stderr:188796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:06:59.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-13T02:06:59.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-13T02:06:59.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 188795 2026-02-13T02:06:59.917 INFO:tasks.workunit.client.0.trial112.stderr:t.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' 4 2026-02-13T02:06:59.917 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-13T02:06:59.918 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-13T02:06:59.918 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-13T02:06:59.918 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-13T02:06:59.918 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-13T02:06:59.918 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-13T02:06:59.918 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-13T02:06:59.918 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-13T02:06:59.918 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-13T02:06:59.918 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-13T02:06:59.918 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-13T02:06:59.918 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-13T02:06:59.918 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-13T02:06:59.918 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:06:59.919 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:06:59.919 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:06:59.919 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:06:59.919 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:06:59.919 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:06:59.919 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:06:59.919 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:06:59.919 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:06:59.919 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:06:59.919 INFO:tasks.workunit.client.0.trial112.stderr:188798: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:06:59.919 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:06:59.919 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:06:59.920 INFO:tasks.workunit.client.0.trial112.stderr:188798: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:06:59.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-13T02:06:59.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-13T02:06:59.921 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-13T02:06:59.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-13T02:06:59.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:257: corrupt_and_repair_two: return_code=0 2026-02-13T02:06:59.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:258: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-13T02:06:59.921 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: get_pg ecpool SOMETHING 2026-02-13T02:06:59.921 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-13T02:06:59.922 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-13T02:06:59.922 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-13T02:06:59.922 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T02:07:00.079 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: local pg=2.0 2026-02-13T02:07:00.079 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:264: corrupt_and_repair_two: repair 2.0 2026-02-13T02:07:00.079 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-13T02:07:00.080 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-13T02:07:00.080 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:07:00.080 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:07:00.080 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:07:00.080 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:07:00.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-13T02:06:39.990298+0000 2026-02-13T02:07:00.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-13T02:07:00.401 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-13T02:07:00.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-13T02:06:39.990298+0000 2026-02-13T02:07:00.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-13T02:07:00.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:06:39.990298+0000 2026-02-13T02:07:00.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:07:00.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:07:00.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:07:00.413 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:07:00.413 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:07:00.413 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:07:00.414 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:07:00.414 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:07:00.575 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:06:39.990298+0000 '>' 2026-02-13T02:06:39.990298+0000 2026-02-13T02:07:00.575 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:07:01.576 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:07:01.576 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:07:01.577 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:07:01.577 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:07:01.577 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:07:01.578 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:07:01.578 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:07:01.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:06:39.990298+0000 '>' 2026-02-13T02:06:39.990298+0000 2026-02-13T02:07:01.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:07:02.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:07:02.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:07:02.743 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:07:02.743 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:07:02.743 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:07:02.743 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:07:02.743 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:07:02.908 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:06:39.990298+0000 '>' 2026-02-13T02:06:39.990298+0000 2026-02-13T02:07:02.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:07:03.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:07:03.910 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:07:03.910 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:07:03.911 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:07:03.911 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:07:03.911 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:07:03.911 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:07:04.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:06:39.990298+0000 '>' 2026-02-13T02:06:39.990298+0000 2026-02-13T02:07:04.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:07:05.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:07:05.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:07:05.078 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:07:05.078 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:07:05.078 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:07:05.079 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:07:05.079 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:07:05.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:07:01.259676+0000 '>' 2026-02-13T02:06:39.990298+0000 2026-02-13T02:07:05.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:07:05.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: corrupt_and_repair_two: pids= 2026-02-13T02:07:05.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-13T02:07:05.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-13T02:07:05.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-13T02:07:05.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 192595"' 2026-02-13T02:07:05.245 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 192595' 2026-02-13T02:07:05.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-13T02:07:05.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-13T02:07:05.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-13T02:07:05.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 192597"' 2026-02-13T02:07:05.246 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 192597' 2026-02-13T02:07:05.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-13T02:07:05.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:271: corrupt_and_repair_two: wait_background pids 2026-02-13T02:07:05.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 192595 192597' 2026-02-13T02:07:05.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-13T02:07:05.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-13T02:07:05.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 192595 2026-02-13T02:07:05.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/192598: /' 2026-02-13T02:07:05.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-13T02:07:05.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/192600: /' 2026-02-13T02:07:05.758 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:07:05.758 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:07:05.758 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-13T02:07:05.758 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:07:05.758 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-13T02:07:05.758 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:07:05.758 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:07:05.758 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-13T02:07:05.758 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:07:05.758 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-13T02:07:05.759 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:07:05.759 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:07:05.759 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:07:05.759 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:07:05.759 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:07:05.759 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:07:05.759 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-13T02:07:05.759 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:07:05.759 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:07:05.759 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-13T02:07:05.759 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:07:05.759 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:07:05.760 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2026-02-13T02:07:05.760 INFO:tasks.workunit.client.0.trial112.stderr:192598: Error getting attr on : 2.0s0_head,0#-4:00000000:::deep_scrub_2.0s0:head#, (61) No data available 2026-02-13T02:07:05.760 INFO:tasks.workunit.client.0.trial112.stderr:192598: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2026-02-13T02:07:05.760 INFO:tasks.workunit.client.0.trial112.stderr:192598: _ 2026-02-13T02:07:05.760 INFO:tasks.workunit.client.0.trial112.stderr:192598: hinfo_key 2026-02-13T02:07:05.760 INFO:tasks.workunit.client.0.trial112.stderr:192598: snapset 2026-02-13T02:07:05.760 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-13T02:07:05.760 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:07:05.760 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:07:05.760 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-13T02:07:05.760 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:07:05.760 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:07:05.761 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:07:05.761 INFO:tasks.workunit.client.0.trial112.stderr:19192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:07:05.761 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:07:05.762 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2026-02-13T02:07:05.762 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:07:05.762 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-13T02:07:05.762 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:07:05.762 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:07:05.762 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2026-02-13T02:07:05.762 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:07:05.762 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2026-02-13T02:07:05.762 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:07:05.762 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:07:05.762 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:07:05.762 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:07:05.763 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:07:05.763 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:07:05.763 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-13T02:07:05.763 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:07:05.763 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:07:05.763 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2026-02-13T02:07:05.763 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:07:05.763 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2026-02-13T02:07:05.763 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2026-02-13T02:07:05.763 INFO:tasks.workunit.client.0.trial112.stderr:192600: _ 2026-02-13T02:07:05.763 INFO:tasks.workunit.client.0.trial112.stderr:192600: hinfo_key 2026-02-13T02:07:05.763 INFO:tasks.workunit.client.0.trial112.stderr:192600: snapset 2026-02-13T02:07:05.764 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2026-02-13T02:07:05.764 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:07:05.764 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:07:05.764 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2026-02-13T02:07:05.764 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:07:05.764 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2026-02-13T02:07:05.764 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:07:05.764 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:07:05.764 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper2598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:07:05.764 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:07:05.764 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:07:05.764 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-13T02:07:05.765 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-13T02:07:05.765 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:07:05.765 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:07:05.765 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:07:05.765 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:07:05.765 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:07:05.765 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:07:05.765 INFO:tasks.workunit.client.0.trial112.stderr:192598: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:07:05.765 INFO:tasks.workunit.client.0.trial112.stderr:192598: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:07:05.765 INFO:tasks.workunit.client.0.trial112.stderr:192598: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:07:05.765 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:07:05.765 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:07:05.765 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:07:05.766 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:07:05.766 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:07:05.766 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:07:05.766 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:07:05.766 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:07:05.766 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:07:05.766 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:07:05.766 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:07:05.766 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-13T02:07:05.766 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-13T02:07:05.766 INFO:tasks.workunit.client.0.trial112.stderr:192598: start osd.3 2026-02-13T02:07:05.767 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.s.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:07:05.767 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:07:05.767 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2026-02-13T02:07:05.767 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2026-02-13T02:07:05.767 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:07:05.768 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:07:05.768 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:07:05.768 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:07:05.768 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:07:05.768 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:07:05.768 INFO:tasks.workunit.client.0.trial112.stderr:192600: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:07:05.769 INFO:tasks.workunit.client.0.trial112.stderr:192600: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:07:05.769 INFO:tasks.workunit.client.0.trial112.stderr:192600: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:07:05.769 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:07:05.769 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:07:05.769 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:07:05.769 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:07:05.769 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:07:05.770 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:07:05.770 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:07:05.770 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:07:05.770 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:07:05.770 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:07:05.771 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:07:05.771 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2026-02-13T02:07:05.771 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2026-02-13T02:07:05.771 INFO:tasks.workunit.client.0.trial112.stderr:192600: start osd.5 2026-02-13T02:07:07.251 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:07:07.251 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2026-02-13T02:07:07.251 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2026-02-13T02:07:07.251 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:07:07.251 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:07:07.251 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:07:07.251 INFO:tasks.workunit.client.0.trial112.stderr:192600: 2026-02-13T02:07:05.764+0000 7fed9b598a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:07.252 INFO:tasks.workunit.client.0.trial112.stderr:192600: 2026-02-13T02:07:05.768+0000 7fed9b598a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:07.252 INFO:tasks.workunit.client.0.trial112.stderr:192600: 2026-02-13T02:07:05.768+0000 7fed9b598a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:07.252 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2026-02-13T02:07:07.252 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:07:07.252 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-13T02:07:07.252 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:07:07.252 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:07:07.252 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:07:07.252 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:07:07.252 INFO:tasks.workunit.client.0.trial112.stderr:192600: 0 2026-02-13T02:07:07.252 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:07:07.252 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-13T02:07:07.252 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:07:07.253 INFO:tasks.workunit.client.0.trial112.stderr:192600: 2026-02-13T02:07:06.236+0000 7fed9b598a40 -1 Falling back to public interface 2026-02-13T02:07:07.253 INFO:tasks.workunit.client.0.trial112.stderr:192600: 2026-02-13T02:07:06.928+0000 7fed9b598a40 -1 osd.5 102 log_to_monitors true 2026-02-13T02:07:07.253 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:07:07.253 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:07:07.253 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:07:07.253 INFO:tasks.workunit.client.0.trial112.stderr:192600: 1 2026-02-13T02:07:07.253 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:07:07.253 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-13T02:07:07.253 INFO:tasks.workunit.client.0.trial112.stderr:192600: osd.5 up in weight 1 up_from 106 up_thru 106 down_at 103 last_clean_interval [100,102) [v2:127.0.0.1:6826/4168348426,v1:127.0.0.1:6827/4168348426] [v2:127.0.0.1:6828/4168348426,v1:127.0.0.1:6829/4168348426] exists,up 670b1f5b-3687-4794-8a8a-767a3a2770e7 2026-02-13T02:07:07.253 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:07:07.253 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:07:07.253 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:07:07.254 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:07:07.254 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:07:07.254 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:07:07.254 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:07:07.254 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:07:07.254 INFO:tasks.workunit.client.0.trial112.stderr:192600: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:07:07.255 INFO:tasks.workunit.client.0.trial112.stderr:192600: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helog' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:07:07.255 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-13T02:07:07.256 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-13T02:07:07.256 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:07:07.256 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:07:07.256 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:07:07.256 INFO:tasks.workunit.client.0.trial112.stderr:192598: 2026-02-13T02:07:05.764+0000 7fc8c71dca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:07.256 INFO:tasks.workunit.client.0.trial112.stderr:192598: 2026-02-13T02:07:05.768+0000 7fc8c71dca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:07.256 INFO:tasks.workunit.client.0.trial112.stderr:192598: 2026-02-13T02:07:05.768+0000 7fc8c71dca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:07.256 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-13T02:07:07.256 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:07:07.256 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-13T02:07:07.256 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:07:07.256 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:07:07.257 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:07:07.257 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:07:07.257 INFO:tasks.workunit.client.0.trial112.stderr:192598: 0 2026-02-13T02:07:07.257 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:07:07.257 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:07:07.257 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:07:07.257 INFO:tasks.workunit.client.0.trial112.stderr:192598: 2026-02-13T02:07:06.272+0000 7fc8c71dca40 -1 Falling back to public interface 2026-02-13T02:07:07.257 INFO:tasks.workunit.client.0.trial112.stderr:192598: 2026-02-13T02:07:06.960+0000 7fc8c71dca40 -1 osd.3 102 log_to_monitors true 2026-02-13T02:07:07.257 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:07:07.257 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:07:07.257 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:07:07.257 INFO:tasks.workunit.client.0.trial112.stderr:192598: 1 2026-02-13T02:07:07.257 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:07:07.258 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:07:07.258 INFO:tasks.workunit.client.0.trial112.stderr:192598: osd.3 up in weight 1 up_from 106 up_thru 106 down_at 103 last_clean_interval [101,102) [v2:127.0.0.1:6842/2561676697,v1:127.0.0.1:6843/2561676697] [v2:127.0.0.1:6844/2561676697,v1:127.0.0.1:6845/2561676697] exists,up ad30d011-cd62-4310-8456-c2d78faa8a93 2026-02-13T02:07:07.258 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:07:07.258 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:07:07.258 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:07:07.258 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:07:07.258 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:07:07.258 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:07:07.258 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:07:07.815 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeolpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:07:07.815 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:07:07.815 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:07:07.815 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:07:07.815 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:07:07.816 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:07:07.816 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:07:07.816 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:07:07.816 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:07:07.816 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:07:07.816 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:07:07.816 INFO:tasks.workunit.client.0.trial112.stderr:192600: 1 2026-02-13T02:07:07.816 INFO:tasks.workunit.client.0.trial112.stderr:192600: 2 2026-02-13T02:07:07.816 INFO:tasks.workunit.client.0.trial112.stderr:192600: 3 2026-02-13T02:07:07.816 INFO:tasks.workunit.client.0.trial112.stderr:192600: 4 2026-02-13T02:07:07.816 INFO:tasks.workunit.client.0.trial112.stderr:192600: 5 2026-02-13T02:07:07.816 INFO:tasks.workunit.client.0.trial112.stderr:192600: 6 2026-02-13T02:07:07.817 INFO:tasks.workunit.client.0.trial112.stderr:192600: 7 2026-02-13T02:07:07.817 INFO:tasks.workunit.client.0.trial112.stderr:192600: 8 2026-02-13T02:07:07.817 INFO:tasks.workunit.client.0.trial112.stderr:192600: 9' 2026-02-13T02:07:07.817 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:07:07.817 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:07:07.817 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:07:07.817 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836517 2026-02-13T02:07:07.817 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836517 2026-02-13T02:07:07.817 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836517' 2026-02-13T02:07:07.817 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:07:07.817 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:07:07.817 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705700 2026-02-13T02:07:07.817 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705700 2026-02-13T02:07:07.818 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836517 1-38654705700' 2026-02-13T02:07:07.818 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:07:07.818 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:07:07.818 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574884 2026-02-13T02:07:07.818 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574884 2026-02-13T02:07:07.818 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836517 1-38654705700 2-55834574884' 2026-02-13T02:07:07.818 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:07:07.818 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:07:07.818 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=455266533378 2026-02-13T02:07:07.819 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 455266533378 2026-02-13T02:07:07.819 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836517 1-38654705700 2-55834574884 3-455266533378' 2026-02-13T02:07:07.819 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:07:07.828 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-ut_delays 90 .1 2026-02-13T02:07:07.829 INFO:tasks.workunit.client.0.trial112.stderr:192598: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:07:07.829 INFO:tasks.workunit.client.0.trial112.stderr:192598: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:07:07.829 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:07:07.829 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:07:07.829 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:07:07.829 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:07:07.829 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:07:07.829 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:07:07.829 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:07:07.829 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:07:07.829 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:07:07.830 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:07:07.830 INFO:tasks.workunit.client.0.trial112.stderr:192598: 1 2026-02-13T02:07:07.830 INFO:tasks.workunit.client.0.trial112.stderr:192598: 2 2026-02-13T02:07:07.830 INFO:tasks.workunit.client.0.trial112.stderr:192598: 3 2026-02-13T02:07:07.830 INFO:tasks.workunit.client.0.trial112.stderr:192598: 4 2026-02-13T02:07:07.830 INFO:tasks.workunit.client.0.trial112.stderr:192598: 5 2026-02-13T02:07:07.830 INFO:tasks.workunit.client.0.trial112.stderr:192598: 6 2026-02-13T02:07:07.830 INFO:tasks.workunit.client.0.trial112.stderr:192598: 7 2026-02-13T02:07:07.830 INFO:tasks.workunit.client.0.trial112.stderr:192598: 8 2026-02-13T02:07:07.830 INFO:tasks.workunit.client.0.trial112.stderr:192598: 9' 2026-02-13T02:07:07.830 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:07:07.830 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:07:07.830 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:07:07.831 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836518 2026-02-13T02:07:07.831 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836518 2026-02-13T02:07:07.831 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518' 2026-02-13T02:07:07.831 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:07:07.831 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:07:07.831 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705701 2026-02-13T02:07:07.831 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705701 2026-02-13T02:07:07.831 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-38654705701' 2026-02-13T02:07:07.831 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:07:07.831 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:07:07.831 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574885 2026-02-13T02:07:07.831 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574885 2026-02-13T02:07:07.832 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-38654705701 2-55834574885' 2026-02-13T02:07:07.832 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:07:07.832 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:07:07.832 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=455266533379 2026-02-13T02:07:07.832 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 455266533379 2026-02-13T02:07:08.266 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-38654705701 2-55834574885 3-455266533379' 2026-02-13T02:07:08.266 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:07:08.266 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-13T02:07:08.267 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280547 2026-02-13T02:07:08.267 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280547 2026-02-13T02:07:08.267 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-38654705701 2-55834574885 3-455266533379 4-94489280547' 2026-02-13T02:07:08.267 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:07:08.267 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-13T02:07:08.267 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=455266533379 2026-02-13T02:07:08.268 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 455266533379 2026-02-13T02:07:08.268 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-38654705701 2-55834574885 3-455266533379 4-94489280547 5-455266533379' 2026-02-13T02:07:08.268 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:07:08.268 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-13T02:07:08.268 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953505 2026-02-13T02:07:08.269 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953505 2026-02-13T02:07:08.269 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-38654705701 2-55834574885 3-455266533379 4-94489280547 5-455266533379 6-137438953505' 2026-02-13T02:07:08.269 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:07:08.269 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-13T02:07:08.269 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789984 2026-02-13T02:07:08.269 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789984 2026-02-13T02:07:08.270 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-38654705701 2-55834574885 3-455266533379 4-94489280547 5-455266533379 6-137438953505 7-158913789984' 2026-02-13T02:07:08.270 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:07:08.270 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-13T02:07:08.270 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626463 2026-02-13T02:07:08.270 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626463 2026-02-13T02:07:08.271 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-38654705701 2-55834574885 3-455266533379 4-94489280547 5-455266533379 6-137438953505 7-158913789984 8-180388626463' 2026-02-13T02:07:08.271 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:07:08.271 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-13T02:07:08.271 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=408021893130 2026-02-13T02:07:08.271 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 408021893130 2026-02-13T02:07:08.272 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-38654705701 2-55834574885 3-455266533379 4-944helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-13T02:07:08.272 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280546 2026-02-13T02:07:08.272 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280546 2026-02-13T02:07:08.273 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836517 1-38654705700 2-55834574884 3-455266533378 4-94489280546' 2026-02-13T02:07:08.273 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:07:08.273 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-13T02:07:08.273 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=455266533378 2026-02-13T02:07:08.273 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 455266533378 2026-02-13T02:07:08.273 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836517 1-38654705700 2-55834574884 3-455266533378 4-94489280546 5-455266533378' 2026-02-13T02:07:08.274 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:07:08.274 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-13T02:07:08.274 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953506 2026-02-13T02:07:08.274 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953506 2026-02-13T02:07:08.274 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836517 1-38654705700 2-55834574884 3-455266533378 4-94489280546 5-455266533378 6-137438953506' 2026-02-13T02:07:08.274 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:07:08.275 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-13T02:07:08.275 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789985 2026-02-13T02:07:08.275 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789985 2026-02-13T02:07:08.275 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836517 1-38654705700 2-55834574884 3-455266533378 4-94489280546 5-455266533378 6-137438953506 7-158913789985' 2026-02-13T02:07:08.275 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:07:08.275 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-13T02:07:08.276 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626464 2026-02-13T02:07:08.276 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626464 2026-02-13T02:07:08.276 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836517 1-38654705700 2-55834574884 3-455266533378 4-94489280546 5-455266533378 6-137438953506 7-158913789985 8-180388626464' 2026-02-13T02:07:08.276 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:07:08.276 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-13T02:07:08.276 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=408021893131 2026-02-13T02:07:08.277 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 408021893131 2026-02-13T02:07:08.277 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836517 1-38654705700 2-55834574884 3-455266533378 4-94489280546 5-455266533378 6-137438953506 7-158913789985 8-180388626464 9-408021893131' 2026-02-13T02:07:08.277 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:07:09.925 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836517 2026-02-13T02:07:09.925 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:07:09.925 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:07:09.925 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836517 2026-02-13T02:07:09.925 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:07:09.925 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836517 2026-02-13T02:07:09.926 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836517' 2026-02-13T02:07:09.926 INFO:tasks.workunit.client.0.trial112.stderr:192600: waiting osd.0 seq 21474836517 2026-02-13T02:07:09.926 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:07:09.926 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836516 -lt 21474836517 2026-02-13T02:07:09.926 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:07:09.926 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:07:09.926 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:07:09.926 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836518 -lt 21474836517 2026-02-13T02:07:09.926 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:07:09.926 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705700 2026-02-13T02:07:09.926 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:07:09.927 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:07:09.927 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705700 2026-02-13T02:07:09.927 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:07:09.927 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705700 2026-02-13T02:07:09.927 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705700' 2026-02-13T02:07:09.927 INFO:tasks.workunit.client.0.trial112.stderr:192600: waiting osd.1 seq 38654705700 2026-02-13T02:07:09.927 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:07:09.927 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705702 -lt 38654705700 2026-02-13T02:07:09.927 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:07:09.927 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574884 2026-02-13T02:07:09.927 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:07:09.927 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:07:09.927 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574884 2026-02-13T02:07:09.928 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:07:09.928 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574884 2026-02-13T02:07:09.928 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574884' 2026-02-13T02:07:09.928 INFO:tasks.workunit.client.0.trial112.stderr:192600: waiting osd.2 seq 55834574884 2026-02-13T02:07:09.928 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:07:09.928 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574885 -lt 55834574884 2026-02-13T02:07:09.937 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:22689280547 5-455266533379 6-137438953505 7-158913789984 8-180388626463 9-408021893130' 2026-02-13T02:07:09.938 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:07:09.938 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836518 2026-02-13T02:07:09.938 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:07:09.938 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:07:09.938 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836518 2026-02-13T02:07:09.938 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:07:09.938 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836518 2026-02-13T02:07:09.938 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836518' 2026-02-13T02:07:09.938 INFO:tasks.workunit.client.0.trial112.stderr:192598: waiting osd.0 seq 21474836518 2026-02-13T02:07:09.938 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:07:09.938 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836516 -lt 21474836518 2026-02-13T02:07:09.938 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:07:09.939 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:07:09.939 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:07:09.939 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836518 -lt 21474836518 2026-02-13T02:07:09.939 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:07:09.939 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705701 2026-02-13T02:07:09.939 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:07:09.939 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:07:09.939 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705701 2026-02-13T02:07:09.939 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:07:09.939 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705701 2026-02-13T02:07:09.939 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705701' 2026-02-13T02:07:09.939 INFO:tasks.workunit.client.0.trial112.stderr:192598: waiting osd.1 seq 38654705701 2026-02-13T02:07:09.939 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:07:09.940 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705702 -lt 38654705701 2026-02-13T02:07:09.940 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:07:09.940 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574885 2026-02-13T02:07:09.940 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:07:09.940 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:07:09.940 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574885 2026-02-13T02:07:09.940 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:07:09.940 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574885 2026-02-13T02:07:09.940 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574885' 2026-02-13T02:07:09.940 INFO:tasks.workunit.client.0.trial112.stderr:192598: waiting osd.2 seq 55834574885 2026-02-13T02:07:09.940 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:07:10.422 INFO:tasks.workunit.client.0.trial112.stderr:196: flush_pg_stats: for s in $seqs 2026-02-13T02:07:10.423 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-455266533378 2026-02-13T02:07:10.423 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:07:10.423 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:07:10.423 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-455266533378 2026-02-13T02:07:10.423 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:07:10.423 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=455266533378 2026-02-13T02:07:10.423 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 455266533378' 2026-02-13T02:07:10.423 INFO:tasks.workunit.client.0.trial112.stderr:192600: waiting osd.3 seq 455266533378 2026-02-13T02:07:10.423 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:07:10.424 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 455266533379 -lt 455266533378 2026-02-13T02:07:10.424 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:07:10.424 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-94489280546 2026-02-13T02:07:10.424 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:07:10.424 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-13T02:07:10.424 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-94489280546 2026-02-13T02:07:10.424 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:07:10.424 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280546 2026-02-13T02:07:10.424 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 94489280546' 2026-02-13T02:07:10.424 INFO:tasks.workunit.client.0.trial112.stderr:192600: waiting osd.4 seq 94489280546 2026-02-13T02:07:10.424 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-13T02:07:10.424 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280547 -lt 94489280546 2026-02-13T02:07:10.425 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:07:10.425 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-455266533378 2026-02-13T02:07:10.425 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:07:10.425 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-13T02:07:10.425 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-455266533378 2026-02-13T02:07:10.425 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:07:10.425 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=455266533378 2026-02-13T02:07:10.425 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 455266533378' 2026-02-13T02:07:10.425 INFO:tasks.workunit.client.0.trial112.stderr:192600: waiting osd.5 seq 455266533378 2026-02-13T02:07:10.425 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-13T02:07:10.425 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 455266533379 -lt 455266533378 2026-02-13T02:07:10.425 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:07:10.426 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953506 2026-02-13T02:07:10.426 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:07:10.426 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-13T02:07:10.445 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa2598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574885 -lt 55834574885 2026-02-13T02:07:10.445 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:07:10.445 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-455266533379 2026-02-13T02:07:10.445 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:07:10.445 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:07:10.445 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-455266533379 2026-02-13T02:07:10.445 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:07:10.445 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=455266533379 2026-02-13T02:07:10.445 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 455266533379' 2026-02-13T02:07:10.446 INFO:tasks.workunit.client.0.trial112.stderr:192598: waiting osd.3 seq 455266533379 2026-02-13T02:07:10.446 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:07:10.446 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 455266533379 -lt 455266533379 2026-02-13T02:07:10.446 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:07:10.446 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-94489280547 2026-02-13T02:07:10.446 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:07:10.446 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-13T02:07:10.446 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-94489280547 2026-02-13T02:07:10.446 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:07:10.446 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280547 2026-02-13T02:07:10.446 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 94489280547' 2026-02-13T02:07:10.446 INFO:tasks.workunit.client.0.trial112.stderr:192598: waiting osd.4 seq 94489280547 2026-02-13T02:07:10.446 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-13T02:07:10.447 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280547 -lt 94489280547 2026-02-13T02:07:10.447 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:07:10.447 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-455266533379 2026-02-13T02:07:10.447 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:07:10.447 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-13T02:07:10.447 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-455266533379 2026-02-13T02:07:10.447 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:07:10.447 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=455266533379 2026-02-13T02:07:10.447 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 455266533379' 2026-02-13T02:07:10.447 INFO:tasks.workunit.client.0.trial112.stderr:192598: waiting osd.5 seq 455266533379 2026-02-13T02:07:10.447 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-13T02:07:10.447 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 455266533379 -lt 455266533379 2026-02-13T02:07:10.448 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:07:10.448 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953505 2026-02-13T02:07:10.923 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalon/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953506 2026-02-13T02:07:10.923 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:07:10.923 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953506 2026-02-13T02:07:10.923 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953506' 2026-02-13T02:07:10.923 INFO:tasks.workunit.client.0.trial112.stderr:192600: waiting osd.6 seq 137438953506 2026-02-13T02:07:10.924 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-13T02:07:10.924 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953506 -lt 137438953506 2026-02-13T02:07:10.924 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:07:10.924 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789985 2026-02-13T02:07:10.924 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:07:10.924 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-13T02:07:10.924 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789985 2026-02-13T02:07:10.924 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:07:10.924 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789985 2026-02-13T02:07:10.924 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789985' 2026-02-13T02:07:10.924 INFO:tasks.workunit.client.0.trial112.stderr:192600: waiting osd.7 seq 158913789985 2026-02-13T02:07:10.924 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-13T02:07:10.925 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789985 -lt 158913789985 2026-02-13T02:07:10.925 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:07:10.925 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626464 2026-02-13T02:07:10.925 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:07:10.925 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-13T02:07:10.925 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626464 2026-02-13T02:07:10.925 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:07:10.925 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626464 2026-02-13T02:07:10.925 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626464' 2026-02-13T02:07:10.925 INFO:tasks.workunit.client.0.trial112.stderr:192600: waiting osd.8 seq 180388626464 2026-02-13T02:07:10.925 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-13T02:07:10.925 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626464 -lt 180388626464 2026-02-13T02:07:10.926 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:07:10.926 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-408021893131 2026-02-13T02:07:10.926 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:07:10.926 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-13T02:07:10.926 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-408021893131 2026-02-13T02:07:10.926 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:07:10.926 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=408021893131 2026-02-13T02:07:10.954 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting oe/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:07:10.954 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-13T02:07:10.954 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953505 2026-02-13T02:07:10.954 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:07:10.954 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953505 2026-02-13T02:07:10.955 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953505' 2026-02-13T02:07:10.955 INFO:tasks.workunit.client.0.trial112.stderr:192598: waiting osd.6 seq 137438953505 2026-02-13T02:07:10.955 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-13T02:07:10.955 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953506 -lt 137438953505 2026-02-13T02:07:10.955 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:07:10.955 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789984 2026-02-13T02:07:10.955 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:07:10.955 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-13T02:07:10.955 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789984 2026-02-13T02:07:10.955 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:07:10.955 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789984 2026-02-13T02:07:10.955 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789984' 2026-02-13T02:07:10.956 INFO:tasks.workunit.client.0.trial112.stderr:192598: waiting osd.7 seq 158913789984 2026-02-13T02:07:10.956 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-13T02:07:10.956 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789985 -lt 158913789984 2026-02-13T02:07:10.956 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:07:10.956 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626463 2026-02-13T02:07:10.956 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:07:10.956 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-13T02:07:10.956 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626463 2026-02-13T02:07:10.956 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:07:10.956 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626463 2026-02-13T02:07:10.956 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626463' 2026-02-13T02:07:10.956 INFO:tasks.workunit.client.0.trial112.stderr:192598: waiting osd.8 seq 180388626463 2026-02-13T02:07:10.956 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-13T02:07:10.957 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626464 -lt 180388626463 2026-02-13T02:07:10.957 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:07:10.957 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-408021893130 2026-02-13T02:07:10.957 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:07:10.957 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-13T02:07:10.957 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-408021893130 2026-02-13T02:07:10.957 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:07:11.776 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=408021893130 2026-02-13T02:07:11.776 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 408021893130' 2026-02-13T02:07:11.776 INFO:tasks.workunit.client.0.trial112.stderr:192598: waiting osd.9 seq 408021893130 2026-02-13T02:07:11.777 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-13T02:07:11.777 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 408021893131 -lt 408021893130 2026-02-13T02:07:11.777 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:07:11.777 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:07:11.777 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:07:11.777 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:07:11.777 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:07:11.777 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:07:11.777 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:07:11.777 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:07:11.777 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:07:11.778 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:07:11.778 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:07:11.778 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-13T02:07:11.778 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:07:11.778 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:07:11.778 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:07:11.778 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-13T02:07:11.778 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' -1 2026-02-13T02:07:11.778 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-13T02:07:11.778 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=4 2026-02-13T02:07:11.778 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-13T02:07:11.778 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-13T02:07:11.778 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-13T02:07:11.779 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:07:11.779 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:07:11.779 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:07:11.779 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:07:11.779 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:07:11.779 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:07:11.916 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/stsd.9 seq 408021893131' 2026-02-13T02:07:11.917 INFO:tasks.workunit.client.0.trial112.stderr:192600: waiting osd.9 seq 408021893131 2026-02-13T02:07:11.917 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-13T02:07:11.917 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 408021893131 -lt 408021893131 2026-02-13T02:07:11.917 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:07:11.917 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:07:11.917 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:07:11.918 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:07:11.918 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:07:11.918 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:07:11.918 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:07:11.918 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:07:11.918 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:07:11.918 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:07:11.918 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:07:11.919 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-13T02:07:11.919 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:07:11.919 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:07:11.919 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:07:11.919 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-13T02:07:11.919 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' -1 2026-02-13T02:07:11.920 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-13T02:07:11.920 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=4 2026-02-13T02:07:11.920 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-13T02:07:11.920 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-13T02:07:11.920 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-13T02:07:11.920 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:07:11.920 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:07:11.921 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:07:11.921 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:07:11.921 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:07:11.921 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:07:11.921 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:07:12.698 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephteandalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:07:12.698 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-13T02:07:12.698 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:07:12.698 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:07:12.699 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:07:12.699 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-13T02:07:12.699 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' 4 2026-02-13T02:07:12.699 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-13T02:07:12.699 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-13T02:07:12.699 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-13T02:07:12.699 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-13T02:07:12.700 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-13T02:07:12.700 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-13T02:07:12.700 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-13T02:07:12.700 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-13T02:07:12.700 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-13T02:07:12.700 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-13T02:07:12.700 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-13T02:07:12.700 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-13T02:07:12.700 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-13T02:07:12.700 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:07:12.701 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:07:12.701 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:07:12.701 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:07:12.701 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:07:12.701 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:07:12.701 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:07:12.701 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-13T02:07:12.702 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:07:12.702 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:07:12.884 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-13T02:07:12.884 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:07:12.884 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:07:12.885 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:07:12.885 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-13T02:07:12.885 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' 4 2026-02-13T02:07:12.885 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-13T02:07:12.885 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-13T02:07:12.885 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-13T02:07:12.885 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-13T02:07:12.886 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-13T02:07:12.886 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-13T02:07:12.886 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-13T02:07:12.886 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-13T02:07:12.886 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-13T02:07:12.886 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-13T02:07:12.886 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-13T02:07:12.886 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-13T02:07:12.886 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-13T02:07:12.887 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:07:12.887 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:07:12.887 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:07:12.887 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:07:12.887 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:07:12.887 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:07:12.887 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:07:12.888 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-13T02:07:12.888 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:07:12.888 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:07:12.888 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:07:12.888 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-13T02:07:13.840 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' 4 2026-02-13T02:07:13.840 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-13T02:07:13.840 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-13T02:07:13.841 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-13T02:07:13.841 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-13T02:07:13.841 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-13T02:07:13.841 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-13T02:07:13.841 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-13T02:07:13.841 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-13T02:07:13.841 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-13T02:07:13.841 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 2 >= 13 )) 2026-02-13T02:07:13.841 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-13T02:07:13.841 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.4 2026-02-13T02:07:13.841 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-13T02:07:13.841 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:07:13.841 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:07:13.841 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:07:13.841 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:07:13.841 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:07:13.841 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:07:13.841 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:07:13.841 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:07:13.841 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:07:13.841 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:07:13.841 INFO:tasks.workunit.client.0.trial112.stderr:192600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:07:13.841 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:07:13.841 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:07:13.841 INFO:tasks.workunit.client.0.trial112.stderr:192600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:07:13.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-13T02:07:13.861 INFO:tasks.workunit.client.0.trial112.stderr:st/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:07:13.861 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-13T02:07:13.861 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' 4 2026-02-13T02:07:13.861 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-13T02:07:13.861 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-13T02:07:13.861 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-13T02:07:13.861 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-13T02:07:13.861 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-13T02:07:13.861 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-13T02:07:13.861 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-13T02:07:13.861 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-13T02:07:13.861 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-13T02:07:13.861 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 2 >= 13 )) 2026-02-13T02:07:13.861 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-13T02:07:13.861 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.4 2026-02-13T02:07:13.861 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-13T02:07:13.861 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:07:13.861 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:07:13.861 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:07:13.861 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:07:13.861 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:07:13.861 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:07:13.861 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:07:13.861 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:07:13.862 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:07:13.862 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:07:13.862 INFO:tasks.workunit.client.0.trial112.stderr:192598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:07:13.862 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:07:13.862 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:07:13.862 INFO:tasks.workunit.client.0.trial112.stderr:192598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:07:13.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-13T02:07:13.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-13T02:07:13.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 192597 2026-02-13T02:07:13.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-13T02:07:13.862 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-13T02:07:13.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-13T02:07:13.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:272: corrupt_and_repair_two: return_code=0 2026-02-13T02:07:13.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-13T02:07:13.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:275: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-13T02:07:13.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:276: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-13T02:07:13.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-13T02:07:13.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:07:13.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:07:13.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:07:13.884 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:07:13.884 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:07:13.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:07:13.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:07:13.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:07:14.199 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:07:14.200 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:07:14.202 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:07:14.202 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:07:14.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:07:14.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:07:14.205 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:07:14.206 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:07:14.206 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:07:14.206 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:07:14.207 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:07:14.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:07:14.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:07:14.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:07:14.242 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:07:14.242 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:07:14.242 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:07:14.242 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:07:14.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:07:14.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:07:14.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-13T02:07:14.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-13T02:07:14.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-13T02:07:14.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-13T02:07:14.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:07:14.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:07:14.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:07:14.244 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:07:14.244 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:07:14.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:07:14.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:07:14.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:07:14.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:07:14.248 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:07:14.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:07:14.249 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:07:14.251 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:07:14.251 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:07:14.251 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:07:14.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:07:14.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:07:14.253 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:07:14.253 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:07:14.254 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:07:14.255 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:07:14.255 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:07:14.257 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:07:14.257 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:07:14.257 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:07:14.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:07:14.258 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:07:14.258 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:07:14.258 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-13T02:07:14.260 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T02:07:14.260 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:07:14.260 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:07:14.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.35207 2026-02-13T02:07:14.262 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T02:07:14.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-13T02:07:14.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T02:07:14.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-13T02:07:14.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_lrc_overwrites td/osd-scrub-repair 2026-02-13T02:07:14.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:905: TEST_corrupt_and_repair_lrc_overwrites: '[' true = true ']' 2026-02-13T02:07:14.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:906: TEST_corrupt_and_repair_lrc_overwrites: corrupt_and_repair_lrc td/osd-scrub-repair true 2026-02-13T02:07:14.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:884: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2026-02-13T02:07:14.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:885: corrupt_and_repair_lrc: local allow_overwrites=true 2026-02-13T02:07:14.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:886: corrupt_and_repair_lrc: local poolname=ecpool 2026-02-13T02:07:14.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:888: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2026-02-13T02:07:14.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-13T02:07:14.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-13T02:07:14.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-13T02:07:14.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-13T02:07:14.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-13T02:07:14.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-13T02:07:14.286 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-13T02:07:14.286 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:07:14.286 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:07:14.287 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:07:14.287 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:07:14.287 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:07:14.287 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:07:14.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-13T02:07:14.315 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-13T02:07:14.316 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-13T02:07:14.316 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:07:14.316 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:07:14.316 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-13T02:07:14.316 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-13T02:07:14.317 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:07:14.317 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:07:14.317 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:07:14.317 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:07:14.317 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:07:14.318 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:07:14.318 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:07:14.318 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:07:14.318 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get fsid 2026-02-13T02:07:14.376 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-13T02:07:14.376 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:07:14.376 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:07:14.376 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-13T02:07:14.377 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-13T02:07:14.377 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:07:14.377 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:07:14.377 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:07:14.378 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:07:14.378 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:07:14.378 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:07:14.378 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:07:14.378 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:07:14.378 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get mon_host 2026-02-13T02:07:14.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:889: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2026-02-13T02:07:14.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-13T02:07:14.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-13T02:07:14.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-13T02:07:14.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-13T02:07:14.435 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-13T02:07:14.435 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-13T02:07:14.531 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-13T02:07:14.531 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:07:14.531 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:07:14.532 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:07:14.532 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:07:14.532 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:07:14.532 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:07:14.533 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:07:14.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:07:14.552 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: seq 0 9 2026-02-13T02:07:14.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-13T02:07:14.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 0 2026-02-13T02:07:14.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:07:14.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:07:14.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-13T02:07:14.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:07:14.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:07:14.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:07:14.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:07:14.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:07:14.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:07:14.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:07:14.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:07:14.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:07:14.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:07:14.556 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:07:14.556 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:07:14.556 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:07:14.556 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:07:14.556 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:07:14.556 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:07:14.556 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:07:14.558 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:07:14.558 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:07:14.558 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:07:14.558 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:07:14.558 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:07:14.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:07:14.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:07:14.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:07:14.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:07:14.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:07:14.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:07:14.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:07:14.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:07:14.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:07:14.559 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:07:14.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3e89934b-394b-49b9-b68e-4abae0d069c6 2026-02-13T02:07:14.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 3e89934b-394b-49b9-b68e-4abae0d069c6' 2026-02-13T02:07:14.560 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 3e89934b-394b-49b9-b68e-4abae0d069c6 2026-02-13T02:07:14.560 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:07:14.568 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBSh45pVhttIRAA9Nm88GUCNvYdQ2YQAE47qQ== 2026-02-13T02:07:14.568 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBSh45pVhttIRAA9Nm88GUCNvYdQ2YQAE47qQ=="}' 2026-02-13T02:07:14.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3e89934b-394b-49b9-b68e-4abae0d069c6 -i td/osd-scrub-repair/0/new.json 2026-02-13T02:07:14.658 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:07:14.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-13T02:07:14.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBSh45pVhttIRAA9Nm88GUCNvYdQ2YQAE47qQ== --osd-uuid 3e89934b-394b-49b9-b68e-4abae0d069c6 2026-02-13T02:07:14.683 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:14.671+0000 7f06ce40ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:14.684 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:14.671+0000 7f06ce40ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:14.685 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:14.671+0000 7f06ce40ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:14.686 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:14.675+0000 7f06ce40ba40 -1 bdev(0x5628a9b3cc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-13T02:07:14.686 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:14.675+0000 7f06ce40ba40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-13T02:07:15.469 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-13T02:07:15.469 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:07:15.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-13T02:07:15.471 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T02:07:15.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:07:15.578 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T02:07:15.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-13T02:07:15.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:07:15.580 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:07:15.580 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:07:15.580 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:07:15.602 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:15.591+0000 7faac0319a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:15.604 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:15.591+0000 7faac0319a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:15.606 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:15.595+0000 7faac0319a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:15.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-13T02:07:15.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:07:15.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:07:15.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:07:15.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:07:15.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:07:15.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:07:15.676 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:07:15.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:07:15.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:07:15.766 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:07:16.103 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:16.091+0000 7faac0319a40 -1 Falling back to public interface 2026-02-13T02:07:16.767 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:07:16.767 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:07:16.767 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:07:16.768 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:07:16.768 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:07:16.768 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:07:16.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:07:17.008 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:16.995+0000 7faac0319a40 -1 osd.0 0 log_to_monitors true 2026-02-13T02:07:17.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:07:17.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:07:17.863 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:07:17.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:07:17.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:07:17.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:07:18.025 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2444962238,v1:127.0.0.1:6801/2444962238] [v2:127.0.0.1:6802/2444962238,v1:127.0.0.1:6803/2444962238] exists,up 3e89934b-394b-49b9-b68e-4abae0d069c6 2026-02-13T02:07:18.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:07:18.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:07:18.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:07:18.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-13T02:07:18.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 1 2026-02-13T02:07:18.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:07:18.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:07:18.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-13T02:07:18.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:07:18.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:07:18.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:07:18.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:07:18.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:07:18.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:07:18.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:07:18.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:07:18.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:07:18.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:07:18.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:07:18.031 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:07:18.031 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:07:18.031 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:07:18.031 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:07:18.031 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:07:18.032 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:07:18.032 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:07:18.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:07:18.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:07:18.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:07:18.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:07:18.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:07:18.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:07:18.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:07:18.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:07:18.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:07:18.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:07:18.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:07:18.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:07:18.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:07:18.033 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:07:18.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d1bd0c5e-eb3d-4cec-9d06-5a3c5d0fda1e 2026-02-13T02:07:18.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 d1bd0c5e-eb3d-4cec-9d06-5a3c5d0fda1e' 2026-02-13T02:07:18.034 INFO:tasks.workunit.client.0.trial112.stdout:add osd1 d1bd0c5e-eb3d-4cec-9d06-5a3c5d0fda1e 2026-02-13T02:07:18.034 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:07:18.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBWh45pKScXAhAAQ/NmlKLd+L1l0fTAzimyQg== 2026-02-13T02:07:18.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBWh45pKScXAhAAQ/NmlKLd+L1l0fTAzimyQg=="}' 2026-02-13T02:07:18.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d1bd0c5e-eb3d-4cec-9d06-5a3c5d0fda1e -i td/osd-scrub-repair/1/new.json 2026-02-13T02:07:18.212 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:07:18.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-13T02:07:18.223 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBWh45pKScXAhAAQ/NmlKLd+L1l0fTAzimyQg== --osd-uuid d1bd0c5e-eb3d-4cec-9d06-5a3c5d0fda1e 2026-02-13T02:07:18.247 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:18.235+0000 7fdd7e034a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:18.248 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:18.235+0000 7fdd7e034a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:18.249 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:18.235+0000 7fdd7e034a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:18.249 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:18.235+0000 7fdd7e034a40 -1 bdev(0x55cd721c2c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-13T02:07:18.250 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:18.239+0000 7fdd7e034a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-13T02:07:18.877 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-13T02:07:18.877 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:07:18.878 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-13T02:07:18.878 INFO:tasks.workunit.client.0.trial112.stdout:adding osd1 key to auth repository 2026-02-13T02:07:18.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:07:19.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-13T02:07:19.092 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T02:07:19.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:07:19.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:07:19.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:07:19.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:07:19.116 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:19.103+0000 7fca8f42ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:19.118 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:19.107+0000 7fca8f42ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:19.119 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:19.107+0000 7fca8f42ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:19.253 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-13T02:07:19.253 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:07:19.253 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:07:19.253 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:07:19.254 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:07:19.254 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:07:19.254 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:07:19.254 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:07:19.255 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:07:19.255 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:07:19.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:07:19.499 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:19.487+0000 7fca8f42ba40 -1 Falling back to public interface 2026-02-13T02:07:20.125 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:20.115+0000 7fca8f42ba40 -1 osd.1 0 log_to_monitors true 2026-02-13T02:07:20.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:07:20.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:07:20.419 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:07:20.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:07:20.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:07:20.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:07:20.584 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:07:21.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:07:21.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:07:21.587 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:07:21.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:07:21.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:07:21.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:07:21.751 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3400591977,v1:127.0.0.1:6811/3400591977] [v2:127.0.0.1:6812/3400591977,v1:127.0.0.1:6813/3400591977] exists,up d1bd0c5e-eb3d-4cec-9d06-5a3c5d0fda1e 2026-02-13T02:07:21.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:07:21.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:07:21.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:07:21.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-13T02:07:21.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 2 2026-02-13T02:07:21.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:07:21.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:07:21.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-13T02:07:21.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:07:21.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-13T02:07:21.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:07:21.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:07:21.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:07:21.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:07:21.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-13T02:07:21.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-13T02:07:21.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:07:21.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:07:21.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:07:21.756 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:07:21.756 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:07:21.756 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:07:21.757 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:07:21.757 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:07:21.757 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:07:21.757 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:07:21.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:07:21.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:07:21.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:07:21.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:07:21.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:07:21.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:07:21.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:07:21.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:07:21.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:07:21.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:07:21.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:07:21.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:07:21.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-13T02:07:21.758 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:07:21.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=056f78d6-7efe-43c7-97b6-87cd542a5c82 2026-02-13T02:07:21.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 056f78d6-7efe-43c7-97b6-87cd542a5c82' 2026-02-13T02:07:21.758 INFO:tasks.workunit.client.0.trial112.stdout:add osd2 056f78d6-7efe-43c7-97b6-87cd542a5c82 2026-02-13T02:07:21.759 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:07:21.769 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBZh45pnDRjLRAA+S+nxHpbAA682TxFkT91dQ== 2026-02-13T02:07:21.769 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBZh45pnDRjLRAA+S+nxHpbAA682TxFkT91dQ=="}' 2026-02-13T02:07:21.769 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 056f78d6-7efe-43c7-97b6-87cd542a5c82 -i td/osd-scrub-repair/2/new.json 2026-02-13T02:07:21.934 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:07:21.944 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-13T02:07:21.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBZh45pnDRjLRAA+S+nxHpbAA682TxFkT91dQ== --osd-uuid 056f78d6-7efe-43c7-97b6-87cd542a5c82 2026-02-13T02:07:21.962 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:21.951+0000 7f8867831a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:21.963 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:21.951+0000 7f8867831a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:21.965 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:21.951+0000 7f8867831a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:21.965 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:21.951+0000 7f8867831a40 -1 bdev(0x55c5fa2dac00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-13T02:07:21.965 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:21.951+0000 7f8867831a40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-13T02:07:22.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-13T02:07:22.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:07:22.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-13T02:07:22.607 INFO:tasks.workunit.client.0.trial112.stdout:adding osd2 key to auth repository 2026-02-13T02:07:22.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:07:22.817 INFO:tasks.workunit.client.0.trial112.stdout:start osd.2 2026-02-13T02:07:22.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-13T02:07:22.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:07:22.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:07:22.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:07:22.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:07:22.842 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:22.831+0000 7f5238591a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:22.844 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:22.831+0000 7f5238591a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:22.845 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:22.831+0000 7f5238591a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:22.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-13T02:07:22.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:07:22.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T02:07:22.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:07:22.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:07:22.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:07:22.976 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:07:22.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:07:22.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:07:22.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:07:23.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:07:23.211 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:23.199+0000 7f5238591a40 -1 Falling back to public interface 2026-02-13T02:07:23.756 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:23.743+0000 7f5238591a40 -1 osd.2 0 log_to_monitors true 2026-02-13T02:07:24.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:07:24.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:07:24.142 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:07:24.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:07:24.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:07:24.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:07:24.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:07:25.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:07:25.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:07:25.310 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:07:25.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:07:25.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:07:25.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:07:25.477 INFO:tasks.workunit.client.0.trial112.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1712218883,v1:127.0.0.1:6819/1712218883] [v2:127.0.0.1:6820/1712218883,v1:127.0.0.1:6821/1712218883] exists,up 056f78d6-7efe-43c7-97b6-87cd542a5c82 2026-02-13T02:07:25.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:07:25.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:07:25.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:07:25.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-13T02:07:25.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 3 2026-02-13T02:07:25.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:07:25.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:07:25.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-13T02:07:25.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:07:25.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:07:25.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:07:25.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:07:25.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:07:25.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:07:25.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-13T02:07:25.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-13T02:07:25.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:07:25.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:07:25.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:07:25.478 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:07:25.478 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:07:25.478 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:07:25.478 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:07:25.478 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:07:25.478 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:07:25.479 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:07:25.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:07:25.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:07:25.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:07:25.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:07:25.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:07:25.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:07:25.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:07:25.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:07:25.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:07:25.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:07:25.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:07:25.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:07:25.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/3 2026-02-13T02:07:25.480 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:07:25.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=eecbad03-21ed-4f9f-89e2-0664db8d736e 2026-02-13T02:07:25.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 eecbad03-21ed-4f9f-89e2-0664db8d736e' 2026-02-13T02:07:25.482 INFO:tasks.workunit.client.0.trial112.stdout:add osd3 eecbad03-21ed-4f9f-89e2-0664db8d736e 2026-02-13T02:07:25.482 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:07:25.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBdh45pMm0RHRAAgeDmWSp7D8a8w9J1RGi2ZQ== 2026-02-13T02:07:25.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBdh45pMm0RHRAAgeDmWSp7D8a8w9J1RGi2ZQ=="}' 2026-02-13T02:07:25.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new eecbad03-21ed-4f9f-89e2-0664db8d736e -i td/osd-scrub-repair/3/new.json 2026-02-13T02:07:25.658 INFO:tasks.workunit.client.0.trial112.stdout:3 2026-02-13T02:07:25.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/3/new.json 2026-02-13T02:07:25.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBdh45pMm0RHRAAgeDmWSp7D8a8w9J1RGi2ZQ== --osd-uuid eecbad03-21ed-4f9f-89e2-0664db8d736e 2026-02-13T02:07:25.695 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:25.683+0000 7fd45ba10a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:25.696 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:25.683+0000 7fd45ba10a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:25.697 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:25.683+0000 7fd45ba10a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:25.698 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:25.687+0000 7fd45ba10a40 -1 bdev(0x562406638c00 td/osd-scrub-repair/3/block) open stat got: (1) Operation not permitted 2026-02-13T02:07:25.698 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:25.687+0000 7fd45ba10a40 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2026-02-13T02:07:26.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2026-02-13T02:07:26.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:07:26.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-13T02:07:26.331 INFO:tasks.workunit.client.0.trial112.stdout:adding osd3 key to auth repository 2026-02-13T02:07:26.331 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:07:26.538 INFO:tasks.workunit.client.0.trial112.stdout:start osd.3 2026-02-13T02:07:26.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-13T02:07:26.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:07:26.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:07:26.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:07:26.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:07:26.561 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:26.547+0000 7f73c29baa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:26.563 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:26.551+0000 7f73c29baa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:26.564 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:26.551+0000 7f73c29baa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:26.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-13T02:07:26.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:07:26.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-13T02:07:26.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:07:26.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:07:26.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:07:26.697 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:07:26.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:07:26.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:07:26.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:07:26.859 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:07:26.903 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:26.891+0000 7f73c29baa40 -1 Falling back to public interface 2026-02-13T02:07:27.491 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:27.479+0000 7f73c29baa40 -1 osd.3 0 log_to_monitors true 2026-02-13T02:07:27.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:07:27.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:07:27.862 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:07:27.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:07:27.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:07:27.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:07:28.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:07:29.037 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:07:29.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:07:29.038 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:07:29.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:07:29.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:07:29.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:07:29.206 INFO:tasks.workunit.client.0.trial112.stdout:osd.3 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1507316376,v1:127.0.0.1:6827/1507316376] [v2:127.0.0.1:6828/1507316376,v1:127.0.0.1:6829/1507316376] exists,up eecbad03-21ed-4f9f-89e2-0664db8d736e 2026-02-13T02:07:29.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:07:29.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:07:29.210 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:07:29.210 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-13T02:07:29.210 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 4 2026-02-13T02:07:29.210 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:07:29.210 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:07:29.210 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2026-02-13T02:07:29.210 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:07:29.210 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/4 2026-02-13T02:07:29.210 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:07:29.210 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:07:29.210 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:07:29.210 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:07:29.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2026-02-13T02:07:29.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2026-02-13T02:07:29.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:07:29.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:07:29.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:07:29.211 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:07:29.211 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:07:29.211 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:07:29.211 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:07:29.211 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:07:29.211 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:07:29.212 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:07:29.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:07:29.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:07:29.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:07:29.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:07:29.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:07:29.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:07:29.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:07:29.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:07:29.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:07:29.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:07:29.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:07:29.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:07:29.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/4 2026-02-13T02:07:29.213 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:07:29.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c426a369-7085-47ff-8b3c-af6d26b6bd7c 2026-02-13T02:07:29.214 INFO:tasks.workunit.client.0.trial112.stdout:add osd4 c426a369-7085-47ff-8b3c-af6d26b6bd7c 2026-02-13T02:07:29.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 c426a369-7085-47ff-8b3c-af6d26b6bd7c' 2026-02-13T02:07:29.214 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:07:29.224 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBhh45pmZ7kDBAA/oLa7vVyBe2ZrH9JVX41pw== 2026-02-13T02:07:29.224 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBhh45pmZ7kDBAA/oLa7vVyBe2ZrH9JVX41pw=="}' 2026-02-13T02:07:29.224 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c426a369-7085-47ff-8b3c-af6d26b6bd7c -i td/osd-scrub-repair/4/new.json 2026-02-13T02:07:29.389 INFO:tasks.workunit.client.0.trial112.stdout:4 2026-02-13T02:07:29.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/4/new.json 2026-02-13T02:07:29.401 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBhh45pmZ7kDBAA/oLa7vVyBe2ZrH9JVX41pw== --osd-uuid c426a369-7085-47ff-8b3c-af6d26b6bd7c 2026-02-13T02:07:29.423 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:29.411+0000 7ff98b76fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:29.425 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:29.411+0000 7ff98b76fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:29.426 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:29.415+0000 7ff98b76fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:29.427 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:29.415+0000 7ff98b76fa40 -1 bdev(0x562dbcddcc00 td/osd-scrub-repair/4/block) open stat got: (1) Operation not permitted 2026-02-13T02:07:29.427 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:29.415+0000 7ff98b76fa40 -1 bluestore(td/osd-scrub-repair/4) _read_fsid unparsable uuid 2026-02-13T02:07:30.065 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/4/keyring 2026-02-13T02:07:30.065 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:07:30.066 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2026-02-13T02:07:30.066 INFO:tasks.workunit.client.0.trial112.stdout:adding osd4 key to auth repository 2026-02-13T02:07:30.066 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:07:30.275 INFO:tasks.workunit.client.0.trial112.stdout:start osd.4 2026-02-13T02:07:30.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2026-02-13T02:07:30.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:07:30.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:07:30.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:07:30.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:07:30.299 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:30.287+0000 7f8d8308ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:30.301 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:30.287+0000 7f8d8308ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:30.302 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:30.291+0000 7f8d8308ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:30.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2026-02-13T02:07:30.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:07:30.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2026-02-13T02:07:30.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:07:30.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:07:30.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:07:30.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:07:30.437 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:07:30.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:07:30.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-13T02:07:30.600 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:07:30.707 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:30.695+0000 7f8d8308ba40 -1 Falling back to public interface 2026-02-13T02:07:31.316 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:31.303+0000 7f8d8308ba40 -1 osd.4 0 log_to_monitors true 2026-02-13T02:07:31.601 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:07:31.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:07:31.602 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:07:31.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:07:31.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:07:31.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-13T02:07:31.760 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:07:32.762 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:07:32.762 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:07:32.763 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:07:32.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:07:32.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:07:32.764 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-13T02:07:32.924 INFO:tasks.workunit.client.0.trial112.stdout:osd.4 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/3791403966,v1:127.0.0.1:6835/3791403966] [v2:127.0.0.1:6836/3791403966,v1:127.0.0.1:6837/3791403966] exists,up c426a369-7085-47ff-8b3c-af6d26b6bd7c 2026-02-13T02:07:32.927 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:07:32.927 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:07:32.928 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:07:32.928 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-13T02:07:32.928 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 5 2026-02-13T02:07:32.928 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:07:32.928 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:07:32.928 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2026-02-13T02:07:32.928 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:07:32.928 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/5 2026-02-13T02:07:32.928 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:07:32.928 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:07:32.929 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:07:32.929 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:07:32.929 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2026-02-13T02:07:32.929 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2026-02-13T02:07:32.929 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:07:32.929 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:07:32.929 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:07:32.929 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:07:32.929 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:07:32.929 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:07:32.929 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:07:32.929 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:07:32.930 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:07:32.930 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:07:32.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:07:32.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:07:32.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:07:32.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:07:32.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:07:32.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:07:32.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:07:32.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:07:32.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:07:32.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:07:32.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:07:32.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:07:32.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/5 2026-02-13T02:07:32.931 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:07:32.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=03eef3ce-a0e3-4fec-86e3-c40fb660505f 2026-02-13T02:07:32.932 INFO:tasks.workunit.client.0.trial112.stdout:add osd5 03eef3ce-a0e3-4fec-86e3-c40fb660505f 2026-02-13T02:07:32.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 03eef3ce-a0e3-4fec-86e3-c40fb660505f' 2026-02-13T02:07:32.933 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:07:32.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBkh45p+fC6NxAA038OEM9fEaFroQIvsDSt6A== 2026-02-13T02:07:32.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBkh45p+fC6NxAA038OEM9fEaFroQIvsDSt6A=="}' 2026-02-13T02:07:32.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 03eef3ce-a0e3-4fec-86e3-c40fb660505f -i td/osd-scrub-repair/5/new.json 2026-02-13T02:07:33.102 INFO:tasks.workunit.client.0.trial112.stdout:5 2026-02-13T02:07:33.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/5/new.json 2026-02-13T02:07:33.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBkh45p+fC6NxAA038OEM9fEaFroQIvsDSt6A== --osd-uuid 03eef3ce-a0e3-4fec-86e3-c40fb660505f 2026-02-13T02:07:33.128 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:33.115+0000 7fd9ac892a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:33.129 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:33.119+0000 7fd9ac892a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:33.130 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:33.119+0000 7fd9ac892a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:33.131 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:33.119+0000 7fd9ac892a40 -1 bdev(0x55d331132c00 td/osd-scrub-repair/5/block) open stat got: (1) Operation not permitted 2026-02-13T02:07:33.131 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:33.119+0000 7fd9ac892a40 -1 bluestore(td/osd-scrub-repair/5) _read_fsid unparsable uuid 2026-02-13T02:07:33.848 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/5/keyring 2026-02-13T02:07:33.848 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:07:33.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2026-02-13T02:07:33.849 INFO:tasks.workunit.client.0.trial112.stdout:adding osd5 key to auth repository 2026-02-13T02:07:33.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:07:34.055 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2026-02-13T02:07:34.056 INFO:tasks.workunit.client.0.trial112.stdout:start osd.5 2026-02-13T02:07:34.057 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:07:34.057 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:07:34.057 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:07:34.058 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:07:34.079 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:34.067+0000 7f89bd120a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:34.081 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:34.067+0000 7f89bd120a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:34.082 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:34.071+0000 7f89bd120a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:34.218 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2026-02-13T02:07:34.218 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:07:34.218 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-13T02:07:34.218 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:07:34.218 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:07:34.218 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:07:34.218 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:07:34.218 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:07:34.219 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:07:34.219 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-13T02:07:34.382 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:07:34.491 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:34.479+0000 7f89bd120a40 -1 Falling back to public interface 2026-02-13T02:07:35.087 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:35.075+0000 7f89bd120a40 -1 osd.5 0 log_to_monitors true 2026-02-13T02:07:35.383 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:07:35.383 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:07:35.384 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:07:35.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:07:35.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:07:35.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-13T02:07:35.547 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:07:36.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:07:36.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:07:36.549 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:07:36.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:07:36.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:07:36.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-13T02:07:36.709 INFO:tasks.workunit.client.0.trial112.stdout:osd.5 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2264745870,v1:127.0.0.1:6843/2264745870] [v2:127.0.0.1:6844/2264745870,v1:127.0.0.1:6845/2264745870] exists,up 03eef3ce-a0e3-4fec-86e3-c40fb660505f 2026-02-13T02:07:36.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:07:36.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:07:36.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:07:36.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-13T02:07:36.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 6 2026-02-13T02:07:36.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:07:36.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:07:36.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2026-02-13T02:07:36.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:07:36.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/6 2026-02-13T02:07:36.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:07:36.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:07:36.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:07:36.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:07:36.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2026-02-13T02:07:36.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2026-02-13T02:07:36.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:07:36.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:07:36.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:07:36.714 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:07:36.714 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:07:36.714 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:07:36.714 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:07:36.715 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:07:36.715 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:07:36.715 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:07:36.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:07:36.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:07:36.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:07:36.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:07:36.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:07:36.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:07:36.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:07:36.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:07:36.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:07:36.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:07:36.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:07:36.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:07:36.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/6 2026-02-13T02:07:36.716 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:07:36.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=71c92f17-52a1-4dbe-9d37-14d95d4a3fc8 2026-02-13T02:07:36.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 71c92f17-52a1-4dbe-9d37-14d95d4a3fc8' 2026-02-13T02:07:36.717 INFO:tasks.workunit.client.0.trial112.stdout:add osd6 71c92f17-52a1-4dbe-9d37-14d95d4a3fc8 2026-02-13T02:07:36.717 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:07:36.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBoh45pXyXIKhAAMyYlMd78Rw+VgL1p3RFOmA== 2026-02-13T02:07:36.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBoh45pXyXIKhAAMyYlMd78Rw+VgL1p3RFOmA=="}' 2026-02-13T02:07:36.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 71c92f17-52a1-4dbe-9d37-14d95d4a3fc8 -i td/osd-scrub-repair/6/new.json 2026-02-13T02:07:36.890 INFO:tasks.workunit.client.0.trial112.stdout:6 2026-02-13T02:07:36.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/6/new.json 2026-02-13T02:07:36.901 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBoh45pXyXIKhAAMyYlMd78Rw+VgL1p3RFOmA== --osd-uuid 71c92f17-52a1-4dbe-9d37-14d95d4a3fc8 2026-02-13T02:07:36.918 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:36.907+0000 7fd3ce08aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:36.920 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:36.907+0000 7fd3ce08aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:36.920 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:36.907+0000 7fd3ce08aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:36.921 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:36.907+0000 7fd3ce08aa40 -1 bdev(0x55ee3519ec00 td/osd-scrub-repair/6/block) open stat got: (1) Operation not permitted 2026-02-13T02:07:36.921 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:36.907+0000 7fd3ce08aa40 -1 bluestore(td/osd-scrub-repair/6) _read_fsid unparsable uuid 2026-02-13T02:07:37.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/6/keyring 2026-02-13T02:07:37.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:07:37.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2026-02-13T02:07:37.606 INFO:tasks.workunit.client.0.trial112.stdout:adding osd6 key to auth repository 2026-02-13T02:07:37.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:07:37.813 INFO:tasks.workunit.client.0.trial112.stdout:start osd.6 2026-02-13T02:07:37.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2026-02-13T02:07:37.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:07:37.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:07:37.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:07:37.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:07:37.837 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:37.823+0000 7fa18f91aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:37.839 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:37.827+0000 7fa18f91aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:37.840 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:37.827+0000 7fa18f91aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:37.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2026-02-13T02:07:37.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:07:37.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2026-02-13T02:07:37.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:07:37.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:07:37.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:07:37.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:07:37.974 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:07:37.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:07:37.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-13T02:07:38.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:07:38.243 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:38.231+0000 7fa18f91aa40 -1 Falling back to public interface 2026-02-13T02:07:38.860 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:38.847+0000 7fa18f91aa40 -1 osd.6 0 log_to_monitors true 2026-02-13T02:07:39.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:07:39.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:07:39.142 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:07:39.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:07:39.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:07:39.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-13T02:07:39.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:07:40.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:07:40.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:07:40.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:07:40.301 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:07:40.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:07:40.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-13T02:07:40.465 INFO:tasks.workunit.client.0.trial112.stdout:osd.6 up in weight 1 up_from 31 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/3223009169,v1:127.0.0.1:6851/3223009169] [v2:127.0.0.1:6852/3223009169,v1:127.0.0.1:6853/3223009169] exists,up 71c92f17-52a1-4dbe-9d37-14d95d4a3fc8 2026-02-13T02:07:40.469 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:07:40.469 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:07:40.469 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:07:40.469 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-13T02:07:40.469 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 7 2026-02-13T02:07:40.469 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:07:40.469 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:07:40.469 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=7 2026-02-13T02:07:40.469 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:07:40.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/7 2026-02-13T02:07:40.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:07:40.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:07:40.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:07:40.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:07:40.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2026-02-13T02:07:40.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2026-02-13T02:07:40.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:07:40.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:07:40.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:07:40.470 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:07:40.470 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:07:40.471 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:07:40.471 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:07:40.471 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:07:40.471 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:07:40.471 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:07:40.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:07:40.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:07:40.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:07:40.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:07:40.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:07:40.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:07:40.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:07:40.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:07:40.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:07:40.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:07:40.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:07:40.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:07:40.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/7 2026-02-13T02:07:40.472 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:07:40.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8546caa8-c419-4f5a-92dd-d82fe705ae23 2026-02-13T02:07:40.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd7 8546caa8-c419-4f5a-92dd-d82fe705ae23' 2026-02-13T02:07:40.473 INFO:tasks.workunit.client.0.trial112.stdout:add osd7 8546caa8-c419-4f5a-92dd-d82fe705ae23 2026-02-13T02:07:40.473 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:07:40.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBsh45pp61XHBAA0kaT78PT6eejZMIdUj3P8w== 2026-02-13T02:07:40.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBsh45pp61XHBAA0kaT78PT6eejZMIdUj3P8w=="}' 2026-02-13T02:07:40.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8546caa8-c419-4f5a-92dd-d82fe705ae23 -i td/osd-scrub-repair/7/new.json 2026-02-13T02:07:40.643 INFO:tasks.workunit.client.0.trial112.stdout:7 2026-02-13T02:07:40.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/7/new.json 2026-02-13T02:07:40.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 7 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBsh45pp61XHBAA0kaT78PT6eejZMIdUj3P8w== --osd-uuid 8546caa8-c419-4f5a-92dd-d82fe705ae23 2026-02-13T02:07:40.672 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:40.659+0000 7f3558609a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:40.673 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:40.659+0000 7f3558609a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:40.674 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:40.663+0000 7f3558609a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:40.675 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:40.663+0000 7f3558609a40 -1 bdev(0x55624587ec00 td/osd-scrub-repair/7/block) open stat got: (1) Operation not permitted 2026-02-13T02:07:40.675 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:40.663+0000 7f3558609a40 -1 bluestore(td/osd-scrub-repair/7) _read_fsid unparsable uuid 2026-02-13T02:07:41.336 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/7/keyring 2026-02-13T02:07:41.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:07:41.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd7 key to auth repository 2026-02-13T02:07:41.338 INFO:tasks.workunit.client.0.trial112.stdout:adding osd7 key to auth repository 2026-02-13T02:07:41.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:07:41.536 INFO:tasks.workunit.client.0.trial112.stdout:start osd.7 2026-02-13T02:07:41.537 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.7 2026-02-13T02:07:41.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 7 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:07:41.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:07:41.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:07:41.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:07:41.552 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:41.539+0000 7fcb4687aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:41.553 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:41.539+0000 7fcb4687aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:41.554 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:41.543+0000 7fcb4687aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:41.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 7 2026-02-13T02:07:41.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:07:41.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=7 2026-02-13T02:07:41.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:07:41.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:07:41.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:07:41.700 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:07:41.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:07:41.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:07:41.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2026-02-13T02:07:41.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:07:41.947 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:41.935+0000 7fcb4687aa40 -1 Falling back to public interface 2026-02-13T02:07:42.569 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:42.555+0000 7fcb4687aa40 -1 osd.7 0 log_to_monitors true 2026-02-13T02:07:42.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:07:42.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:07:42.867 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:07:42.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:07:42.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:07:42.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2026-02-13T02:07:43.024 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:07:44.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:07:44.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:07:44.026 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:07:44.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:07:44.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:07:44.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2026-02-13T02:07:44.187 INFO:tasks.workunit.client.0.trial112.stdout:osd.7 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/1297746896,v1:127.0.0.1:6859/1297746896] [v2:127.0.0.1:6860/1297746896,v1:127.0.0.1:6861/1297746896] exists,up 8546caa8-c419-4f5a-92dd-d82fe705ae23 2026-02-13T02:07:44.190 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:07:44.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:07:44.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:07:44.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-13T02:07:44.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 8 2026-02-13T02:07:44.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:07:44.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:07:44.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=8 2026-02-13T02:07:44.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:07:44.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/8 2026-02-13T02:07:44.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:07:44.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:07:44.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:07:44.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:07:44.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2026-02-13T02:07:44.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2026-02-13T02:07:44.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:07:44.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:07:44.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:07:44.192 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:07:44.192 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:07:44.192 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:07:44.192 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:07:44.193 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:07:44.193 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:07:44.193 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:07:44.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:07:44.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:07:44.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:07:44.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:07:44.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:07:44.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:07:44.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:07:44.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:07:44.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:07:44.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:07:44.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:07:44.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:07:44.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/8 2026-02-13T02:07:44.194 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:07:44.195 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=050a3115-02e7-4487-8fc9-0f44fe3dbfaa 2026-02-13T02:07:44.195 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd8 050a3115-02e7-4487-8fc9-0f44fe3dbfaa' 2026-02-13T02:07:44.195 INFO:tasks.workunit.client.0.trial112.stdout:add osd8 050a3115-02e7-4487-8fc9-0f44fe3dbfaa 2026-02-13T02:07:44.195 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:07:44.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBwh45pk6W3CxAAenZgssGpIZngK105S5p8xQ== 2026-02-13T02:07:44.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBwh45pk6W3CxAAenZgssGpIZngK105S5p8xQ=="}' 2026-02-13T02:07:44.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 050a3115-02e7-4487-8fc9-0f44fe3dbfaa -i td/osd-scrub-repair/8/new.json 2026-02-13T02:07:44.370 INFO:tasks.workunit.client.0.trial112.stdout:8 2026-02-13T02:07:44.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/8/new.json 2026-02-13T02:07:44.381 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 8 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBwh45pk6W3CxAAenZgssGpIZngK105S5p8xQ== --osd-uuid 050a3115-02e7-4487-8fc9-0f44fe3dbfaa 2026-02-13T02:07:44.396 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:44.383+0000 7f013f490a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:44.398 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:44.387+0000 7f013f490a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:44.399 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:44.387+0000 7f013f490a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:44.399 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:44.387+0000 7f013f490a40 -1 bdev(0x56219607ec00 td/osd-scrub-repair/8/block) open stat got: (1) Operation not permitted 2026-02-13T02:07:44.399 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:44.387+0000 7f013f490a40 -1 bluestore(td/osd-scrub-repair/8) _read_fsid unparsable uuid 2026-02-13T02:07:45.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/8/keyring 2026-02-13T02:07:45.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:07:45.069 INFO:tasks.workunit.client.0.trial112.stdout:adding osd8 key to auth repository 2026-02-13T02:07:45.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd8 key to auth repository 2026-02-13T02:07:45.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:07:45.264 INFO:tasks.workunit.client.0.trial112.stdout:start osd.8 2026-02-13T02:07:45.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.8 2026-02-13T02:07:45.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 8 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:07:45.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:07:45.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:07:45.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:07:45.280 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:45.267+0000 7f39fdbeca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:45.281 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:45.271+0000 7f39fdbeca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:45.282 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:45.271+0000 7f39fdbeca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:45.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 8 2026-02-13T02:07:45.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:07:45.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=8 2026-02-13T02:07:45.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:07:45.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:07:45.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:07:45.421 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:07:45.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:07:45.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:07:45.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2026-02-13T02:07:45.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:07:45.659 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:45.647+0000 7f39fdbeca40 -1 Falling back to public interface 2026-02-13T02:07:46.265 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:46.251+0000 7f39fdbeca40 -1 osd.8 0 log_to_monitors true 2026-02-13T02:07:46.585 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:07:46.585 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:07:46.585 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:07:46.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:07:46.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:07:46.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2026-02-13T02:07:46.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:07:47.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:07:47.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:07:47.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:07:47.753 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:07:47.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:07:47.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2026-02-13T02:07:47.913 INFO:tasks.workunit.client.0.trial112.stdout:osd.8 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/1588714594,v1:127.0.0.1:6867/1588714594] [v2:127.0.0.1:6868/1588714594,v1:127.0.0.1:6869/1588714594] exists,up 050a3115-02e7-4487-8fc9-0f44fe3dbfaa 2026-02-13T02:07:47.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:07:47.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:07:47.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:07:47.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-13T02:07:47.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 9 2026-02-13T02:07:47.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:07:47.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:07:47.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=9 2026-02-13T02:07:47.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:07:47.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/9 2026-02-13T02:07:47.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:07:47.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:07:47.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:07:47.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:07:47.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2026-02-13T02:07:47.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2026-02-13T02:07:47.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:07:47.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:07:47.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:07:47.918 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:07:47.918 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:07:47.918 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:07:47.918 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:07:47.918 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:07:47.918 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:07:47.918 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:07:47.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:07:47.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:07:47.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:07:47.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:07:47.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:07:47.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:07:47.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:07:47.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:07:47.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:07:47.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:07:47.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:07:47.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:07:47.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/9 2026-02-13T02:07:47.920 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:07:47.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=18292385-2986-4859-a36e-1a5e00d7be2a 2026-02-13T02:07:47.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd9 18292385-2986-4859-a36e-1a5e00d7be2a' 2026-02-13T02:07:47.921 INFO:tasks.workunit.client.0.trial112.stdout:add osd9 18292385-2986-4859-a36e-1a5e00d7be2a 2026-02-13T02:07:47.921 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:07:47.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBzh45pDaz3NhAAXB7iyeLYhtEVOdW5B70TVg== 2026-02-13T02:07:47.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBzh45pDaz3NhAAXB7iyeLYhtEVOdW5B70TVg=="}' 2026-02-13T02:07:47.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 18292385-2986-4859-a36e-1a5e00d7be2a -i td/osd-scrub-repair/9/new.json 2026-02-13T02:07:48.084 INFO:tasks.workunit.client.0.trial112.stdout:9 2026-02-13T02:07:48.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/9/new.json 2026-02-13T02:07:48.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 9 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBzh45pDaz3NhAAXB7iyeLYhtEVOdW5B70TVg== --osd-uuid 18292385-2986-4859-a36e-1a5e00d7be2a 2026-02-13T02:07:48.122 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:48.111+0000 7f378a5e1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:48.123 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:48.111+0000 7f378a5e1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:48.125 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:48.111+0000 7f378a5e1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:48.125 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:48.111+0000 7f378a5e1a40 -1 bdev(0x55d456ea0c00 td/osd-scrub-repair/9/block) open stat got: (1) Operation not permitted 2026-02-13T02:07:48.126 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:48.115+0000 7f378a5e1a40 -1 bluestore(td/osd-scrub-repair/9) _read_fsid unparsable uuid 2026-02-13T02:07:48.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/9/keyring 2026-02-13T02:07:48.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:07:48.766 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd9 key to auth repository 2026-02-13T02:07:48.767 INFO:tasks.workunit.client.0.trial112.stdout:adding osd9 key to auth repository 2026-02-13T02:07:48.767 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:07:48.973 INFO:tasks.workunit.client.0.trial112.stdout:start osd.9 2026-02-13T02:07:48.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.9 2026-02-13T02:07:48.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 9 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:07:48.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:07:48.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:07:48.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:07:48.998 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:48.987+0000 7f8ac4cdca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:49.000 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:48.987+0000 7f8ac4cdca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:49.001 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:48.987+0000 7f8ac4cdca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:07:49.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 9 2026-02-13T02:07:49.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:07:49.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2026-02-13T02:07:49.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:07:49.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:07:49.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:07:49.132 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:07:49.132 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:07:49.132 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:07:49.132 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-13T02:07:49.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:07:49.399 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:49.387+0000 7f8ac4cdca40 -1 Falling back to public interface 2026-02-13T02:07:50.019 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:07:50.007+0000 7f8ac4cdca40 -1 osd.9 0 log_to_monitors true 2026-02-13T02:07:50.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:07:50.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:07:50.299 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:07:50.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:07:50.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:07:50.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-13T02:07:50.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:07:51.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:07:51.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:07:51.468 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:07:51.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:07:51.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:07:51.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-13T02:07:51.629 INFO:tasks.workunit.client.0.trial112.stdout:osd.9 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/3023502091,v1:127.0.0.1:6875/3023502091] [v2:127.0.0.1:6876/3023502091,v1:127.0.0.1:6877/3023502091] exists,up 18292385-2986-4859-a36e-1a5e00d7be2a 2026-02-13T02:07:51.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:07:51.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:07:51.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:07:51.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:893: corrupt_and_repair_lrc: create_rbd_pool 2026-02-13T02:07:51.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-13T02:07:51.785 INFO:tasks.workunit.client.0.trial112.stderr:pool 'rbd' does not exist 2026-02-13T02:07:51.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-13T02:07:51.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-13T02:07:52.016 INFO:tasks.workunit.client.0.trial112.stderr:pool 'rbd' created 2026-02-13T02:07:52.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:07:53.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-13T02:07:53.291 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: corrupt_and_repair_lrc: wait_for_clean 2026-02-13T02:07:53.291 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:07:53.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:07:53.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:07:53.292 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:07:53.293 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:07:53.293 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:07:53.293 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:07:53.293 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:07:53.293 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:07:53.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:07:53.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:07:53.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:07:53.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:07:53.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:07:53.404 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:07:53.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:07:53.563 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:07:53.563 INFO:tasks.workunit.client.0.trial112.stderr:2 2026-02-13T02:07:53.563 INFO:tasks.workunit.client.0.trial112.stderr:3 2026-02-13T02:07:53.563 INFO:tasks.workunit.client.0.trial112.stderr:4 2026-02-13T02:07:53.563 INFO:tasks.workunit.client.0.trial112.stderr:5 2026-02-13T02:07:53.563 INFO:tasks.workunit.client.0.trial112.stderr:6 2026-02-13T02:07:53.564 INFO:tasks.workunit.client.0.trial112.stderr:7 2026-02-13T02:07:53.564 INFO:tasks.workunit.client.0.trial112.stderr:8 2026-02-13T02:07:53.564 INFO:tasks.workunit.client.0.trial112.stderr:9' 2026-02-13T02:07:53.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:07:53.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:07:53.564 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:07:53.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836489 2026-02-13T02:07:53.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836489 2026-02-13T02:07:53.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489' 2026-02-13T02:07:53.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:07:53.640 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:07:53.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705672 2026-02-13T02:07:53.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705672 2026-02-13T02:07:53.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-38654705672' 2026-02-13T02:07:53.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:07:53.713 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:07:53.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542151 2026-02-13T02:07:53.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542151 2026-02-13T02:07:53.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-38654705672 2-60129542151' 2026-02-13T02:07:53.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:07:53.786 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:07:53.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411335 2026-02-13T02:07:53.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411335 2026-02-13T02:07:53.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-38654705672 2-60129542151 3-77309411335' 2026-02-13T02:07:53.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:07:53.861 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-13T02:07:53.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280518 2026-02-13T02:07:53.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280518 2026-02-13T02:07:53.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-38654705672 2-60129542151 3-77309411335 4-94489280518' 2026-02-13T02:07:53.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:07:53.934 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-13T02:07:54.006 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964116997 2026-02-13T02:07:54.006 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964116997 2026-02-13T02:07:54.006 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-38654705672 2-60129542151 3-77309411335 4-94489280518 5-115964116997' 2026-02-13T02:07:54.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:07:54.007 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-13T02:07:54.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986180 2026-02-13T02:07:54.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986180 2026-02-13T02:07:54.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-38654705672 2-60129542151 3-77309411335 4-94489280518 5-115964116997 6-133143986180' 2026-02-13T02:07:54.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:07:54.083 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-13T02:07:54.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855364 2026-02-13T02:07:54.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855364 2026-02-13T02:07:54.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-38654705672 2-60129542151 3-77309411335 4-94489280518 5-115964116997 6-133143986180 7-150323855364' 2026-02-13T02:07:54.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:07:54.157 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-13T02:07:54.228 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691843 2026-02-13T02:07:54.229 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691843 2026-02-13T02:07:54.229 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-38654705672 2-60129542151 3-77309411335 4-94489280518 5-115964116997 6-133143986180 7-150323855364 8-171798691843' 2026-02-13T02:07:54.229 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:07:54.229 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-13T02:07:54.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=193273528322 2026-02-13T02:07:54.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 193273528322 2026-02-13T02:07:54.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-38654705672 2-60129542151 3-77309411335 4-94489280518 5-115964116997 6-133143986180 7-150323855364 8-171798691843 9-193273528322' 2026-02-13T02:07:54.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:07:54.302 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836489 2026-02-13T02:07:54.303 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:07:54.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:07:54.304 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836489 2026-02-13T02:07:54.304 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:07:54.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836489 2026-02-13T02:07:54.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2026-02-13T02:07:54.306 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836489 2026-02-13T02:07:54.307 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:07:54.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836489 2026-02-13T02:07:54.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:07:55.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:07:55.469 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:07:55.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836489 2026-02-13T02:07:55.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:07:55.640 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705672 2026-02-13T02:07:55.640 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:07:55.642 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:07:55.642 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705672 2026-02-13T02:07:55.643 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:07:55.644 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705672 2026-02-13T02:07:55.644 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705672' 2026-02-13T02:07:55.644 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705672 2026-02-13T02:07:55.645 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:07:55.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705672 -lt 38654705672 2026-02-13T02:07:55.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:07:55.809 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542151 2026-02-13T02:07:55.809 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:07:55.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:07:55.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542151 2026-02-13T02:07:55.812 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:07:55.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542151 2026-02-13T02:07:55.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542151' 2026-02-13T02:07:55.814 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542151 2026-02-13T02:07:55.814 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:07:55.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542151 -lt 60129542151 2026-02-13T02:07:55.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:07:55.983 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-77309411335 2026-02-13T02:07:55.983 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:07:55.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:07:55.985 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-77309411335 2026-02-13T02:07:55.985 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:07:55.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=77309411335 2026-02-13T02:07:55.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 77309411335' 2026-02-13T02:07:55.987 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.3 seq 77309411335 2026-02-13T02:07:55.987 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:07:56.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 77309411335 -lt 77309411335 2026-02-13T02:07:56.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:07:56.146 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-94489280518 2026-02-13T02:07:56.146 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:07:56.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-13T02:07:56.148 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-94489280518 2026-02-13T02:07:56.148 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:07:56.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280518 2026-02-13T02:07:56.150 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.4 seq 94489280518 2026-02-13T02:07:56.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 94489280518' 2026-02-13T02:07:56.151 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-13T02:07:56.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280518 -lt 94489280518 2026-02-13T02:07:56.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:07:56.322 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-115964116997 2026-02-13T02:07:56.322 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:07:56.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-13T02:07:56.324 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-115964116997 2026-02-13T02:07:56.325 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:07:56.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=115964116997 2026-02-13T02:07:56.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 115964116997' 2026-02-13T02:07:56.327 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.5 seq 115964116997 2026-02-13T02:07:56.327 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-13T02:07:56.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 115964116997 -lt 115964116997 2026-02-13T02:07:56.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:07:56.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-133143986180 2026-02-13T02:07:56.489 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:07:56.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-13T02:07:56.491 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-133143986180 2026-02-13T02:07:56.491 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:07:56.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=133143986180 2026-02-13T02:07:56.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 133143986180' 2026-02-13T02:07:56.493 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.6 seq 133143986180 2026-02-13T02:07:56.493 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-13T02:07:56.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 133143986180 -lt 133143986180 2026-02-13T02:07:56.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:07:56.664 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-150323855364 2026-02-13T02:07:56.664 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:07:56.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-13T02:07:56.666 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-150323855364 2026-02-13T02:07:56.666 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:07:56.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855364 2026-02-13T02:07:56.668 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.7 seq 150323855364 2026-02-13T02:07:56.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 150323855364' 2026-02-13T02:07:56.669 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-13T02:07:56.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855364 -lt 150323855364 2026-02-13T02:07:56.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:07:56.832 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-171798691843 2026-02-13T02:07:56.832 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:07:56.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-13T02:07:56.834 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-171798691843 2026-02-13T02:07:56.834 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:07:56.836 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691843 2026-02-13T02:07:56.836 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.8 seq 171798691843 2026-02-13T02:07:56.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 171798691843' 2026-02-13T02:07:56.837 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-13T02:07:57.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691843 -lt 171798691843 2026-02-13T02:07:57.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:07:57.004 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-193273528322 2026-02-13T02:07:57.004 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:07:57.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-13T02:07:57.006 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-193273528322 2026-02-13T02:07:57.006 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:07:57.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=193273528322 2026-02-13T02:07:57.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 193273528322' 2026-02-13T02:07:57.008 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.9 seq 193273528322 2026-02-13T02:07:57.008 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-13T02:07:57.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 193273528323 -lt 193273528322 2026-02-13T02:07:57.179 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:07:57.180 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:07:57.180 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:07:57.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-13T02:07:57.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:07:57.372 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:07:57.372 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:07:57.372 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:07:57.372 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:07:57.372 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:07:57.372 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:07:57.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-13T02:07:57.528 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:07:57.529 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:07:57.529 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:07:57.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-13T02:07:57.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:07:57.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:07:57.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:896: corrupt_and_repair_lrc: create_ec_pool ecpool true k=4 m=2 l=3 plugin=lrc 2026-02-13T02:07:57.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2522: create_ec_pool: local pool_name=ecpool 2026-02-13T02:07:57.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2523: create_ec_pool: shift 2026-02-13T02:07:57.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2524: create_ec_pool: local allow_overwrites=true 2026-02-13T02:07:57.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2525: create_ec_pool: shift 2026-02-13T02:07:57.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2527: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2026-02-13T02:07:57.943 INFO:tasks.workunit.client.0.trial112.stderr:load: isa 2026-02-13T02:07:57.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2529: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2026-02-13T02:07:57.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2026-02-13T02:07:58.248 INFO:tasks.workunit.client.0.trial112.stderr:pool 'ecpool' created 2026-02-13T02:07:58.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:07:59.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2531: create_ec_pool: '[' true = true ']' 2026-02-13T02:07:59.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2532: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2026-02-13T02:07:59.419 INFO:tasks.workunit.client.0.trial112.stderr:set pool 2 allow_ec_overwrites to true 2026-02-13T02:07:59.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2535: create_ec_pool: wait_for_clean 2026-02-13T02:07:59.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:07:59.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:07:59.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:07:59.429 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:07:59.430 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:07:59.430 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:07:59.430 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:07:59.430 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:07:59.430 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:07:59.548 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:07:59.548 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:07:59.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:07:59.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:07:59.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:07:59.549 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:07:59.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:07:59.713 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:07:59.713 INFO:tasks.workunit.client.0.trial112.stderr:2 2026-02-13T02:07:59.713 INFO:tasks.workunit.client.0.trial112.stderr:3 2026-02-13T02:07:59.713 INFO:tasks.workunit.client.0.trial112.stderr:4 2026-02-13T02:07:59.713 INFO:tasks.workunit.client.0.trial112.stderr:5 2026-02-13T02:07:59.713 INFO:tasks.workunit.client.0.trial112.stderr:6 2026-02-13T02:07:59.713 INFO:tasks.workunit.client.0.trial112.stderr:7 2026-02-13T02:07:59.713 INFO:tasks.workunit.client.0.trial112.stderr:8 2026-02-13T02:07:59.713 INFO:tasks.workunit.client.0.trial112.stderr:9' 2026-02-13T02:07:59.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:07:59.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:07:59.714 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:07:59.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836491 2026-02-13T02:07:59.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836491 2026-02-13T02:07:59.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491' 2026-02-13T02:07:59.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:07:59.786 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:07:59.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705674 2026-02-13T02:07:59.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705674 2026-02-13T02:07:59.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705674' 2026-02-13T02:07:59.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:07:59.859 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:07:59.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542154 2026-02-13T02:07:59.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542154 2026-02-13T02:07:59.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705674 2-60129542154' 2026-02-13T02:07:59.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:07:59.931 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:08:00.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411337 2026-02-13T02:08:00.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411337 2026-02-13T02:08:00.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705674 2-60129542154 3-77309411337' 2026-02-13T02:08:00.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:00.007 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-13T02:08:00.082 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280520 2026-02-13T02:08:00.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280520 2026-02-13T02:08:00.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705674 2-60129542154 3-77309411337 4-94489280520' 2026-02-13T02:08:00.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:00.083 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-13T02:08:00.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964116999 2026-02-13T02:08:00.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964116999 2026-02-13T02:08:00.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705674 2-60129542154 3-77309411337 4-94489280520 5-115964116999' 2026-02-13T02:08:00.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:00.156 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-13T02:08:00.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986183 2026-02-13T02:08:00.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986183 2026-02-13T02:08:00.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705674 2-60129542154 3-77309411337 4-94489280520 5-115964116999 6-133143986183' 2026-02-13T02:08:00.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:00.227 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-13T02:08:00.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855366 2026-02-13T02:08:00.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855366 2026-02-13T02:08:00.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705674 2-60129542154 3-77309411337 4-94489280520 5-115964116999 6-133143986183 7-150323855366' 2026-02-13T02:08:00.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:00.301 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-13T02:08:00.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691845 2026-02-13T02:08:00.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691845 2026-02-13T02:08:00.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705674 2-60129542154 3-77309411337 4-94489280520 5-115964116999 6-133143986183 7-150323855366 8-171798691845' 2026-02-13T02:08:00.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:00.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-13T02:08:00.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=193273528324 2026-02-13T02:08:00.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 193273528324 2026-02-13T02:08:00.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705674 2-60129542154 3-77309411337 4-94489280520 5-115964116999 6-133143986183 7-150323855366 8-171798691845 9-193273528324' 2026-02-13T02:08:00.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:00.448 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836491 2026-02-13T02:08:00.448 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:00.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:08:00.449 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836491 2026-02-13T02:08:00.450 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:00.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836491 2026-02-13T02:08:00.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2026-02-13T02:08:00.451 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836491 2026-02-13T02:08:00.452 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:08:00.619 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836490 -lt 21474836491 2026-02-13T02:08:00.619 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:08:01.621 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:08:01.621 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:08:01.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836491 2026-02-13T02:08:01.779 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:01.779 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705674 2026-02-13T02:08:01.779 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:01.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:08:01.781 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705674 2026-02-13T02:08:01.782 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:01.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705674 2026-02-13T02:08:01.783 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705674 2026-02-13T02:08:01.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705674' 2026-02-13T02:08:01.784 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:08:01.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705674 -lt 38654705674 2026-02-13T02:08:01.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:01.949 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542154 2026-02-13T02:08:01.949 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:01.950 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:08:01.951 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542154 2026-02-13T02:08:01.951 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:01.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542154 2026-02-13T02:08:01.953 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542154 2026-02-13T02:08:01.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542154' 2026-02-13T02:08:01.953 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:08:02.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542154 -lt 60129542154 2026-02-13T02:08:02.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:02.114 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-77309411337 2026-02-13T02:08:02.114 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:02.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:08:02.116 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-77309411337 2026-02-13T02:08:02.116 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:02.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=77309411337 2026-02-13T02:08:02.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 77309411337' 2026-02-13T02:08:02.118 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.3 seq 77309411337 2026-02-13T02:08:02.118 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:08:02.290 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 77309411337 -lt 77309411337 2026-02-13T02:08:02.291 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:02.291 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-94489280520 2026-02-13T02:08:02.291 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:02.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-13T02:08:02.293 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-94489280520 2026-02-13T02:08:02.294 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:02.295 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280520 2026-02-13T02:08:02.295 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.4 seq 94489280520 2026-02-13T02:08:02.295 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 94489280520' 2026-02-13T02:08:02.296 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-13T02:08:02.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280520 -lt 94489280520 2026-02-13T02:08:02.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:02.468 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-115964116999 2026-02-13T02:08:02.468 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:02.469 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-13T02:08:02.470 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-115964116999 2026-02-13T02:08:02.470 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:02.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=115964116999 2026-02-13T02:08:02.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 115964116999' 2026-02-13T02:08:02.472 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.5 seq 115964116999 2026-02-13T02:08:02.473 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-13T02:08:02.642 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 115964116999 -lt 115964116999 2026-02-13T02:08:02.642 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:02.643 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-133143986183 2026-02-13T02:08:02.643 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:02.644 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-13T02:08:02.645 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-133143986183 2026-02-13T02:08:02.645 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:02.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=133143986183 2026-02-13T02:08:02.647 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.6 seq 133143986183 2026-02-13T02:08:02.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 133143986183' 2026-02-13T02:08:02.648 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-13T02:08:02.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 133143986183 -lt 133143986183 2026-02-13T02:08:02.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:02.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-150323855366 2026-02-13T02:08:02.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:02.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-13T02:08:02.814 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-150323855366 2026-02-13T02:08:02.814 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:02.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855366 2026-02-13T02:08:02.815 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.7 seq 150323855366 2026-02-13T02:08:02.816 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 150323855366' 2026-02-13T02:08:02.816 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-13T02:08:02.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855366 -lt 150323855366 2026-02-13T02:08:02.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:02.987 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-171798691845 2026-02-13T02:08:02.987 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:02.988 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-13T02:08:02.989 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-171798691845 2026-02-13T02:08:02.989 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:02.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691845 2026-02-13T02:08:02.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 171798691845' 2026-02-13T02:08:02.991 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.8 seq 171798691845 2026-02-13T02:08:02.991 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-13T02:08:03.152 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691846 -lt 171798691845 2026-02-13T02:08:03.152 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:03.153 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-193273528324 2026-02-13T02:08:03.153 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:03.154 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-13T02:08:03.155 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-193273528324 2026-02-13T02:08:03.155 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:03.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=193273528324 2026-02-13T02:08:03.157 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.9 seq 193273528324 2026-02-13T02:08:03.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 193273528324' 2026-02-13T02:08:03.158 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-13T02:08:03.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 193273528325 -lt 193273528324 2026-02-13T02:08:03.339 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:08:03.339 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:08:03.339 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:08:03.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:08:03.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:08:03.543 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:08:03.543 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:08:03.543 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:08:03.543 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:08:03.543 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:08:03.543 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:08:03.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:08:03.708 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:08:03.708 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:08:03.709 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:08:03.908 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:08:03.908 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:08:03.908 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:08:03.908 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2536: create_ec_pool: return 0 2026-02-13T02:08:03.908 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:897: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2026-02-13T02:08:03.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:308: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2026-02-13T02:08:03.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:309: corrupt_and_repair_erasure_coded: local poolname=ecpool 2026-02-13T02:08:03.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:311: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2026-02-13T02:08:03.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:08:03.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-13T02:08:03.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2026-02-13T02:08:03.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:08:03.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:08:03.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:08:04.115 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:08:04.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:08:04.319 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:08:04.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:08:04.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:08:04.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-13T02:08:04.353 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:313: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2026-02-13T02:08:04.353 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2026-02-13T02:08:04.353 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-13T02:08:04.354 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2026-02-13T02:08:04.354 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T02:08:04.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:313: corrupt_and_repair_erasure_coded: local primary=3 2026-02-13T02:08:04.521 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2026-02-13T02:08:04.521 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: sed -e s/3// 2026-02-13T02:08:04.521 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2026-02-13T02:08:04.521 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-13T02:08:04.522 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2026-02-13T02:08:04.522 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-13T02:08:04.684 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-13T02:08:04.684 INFO:tasks.workunit.client.0.trial112.stderr:5 2026-02-13T02:08:04.684 INFO:tasks.workunit.client.0.trial112.stderr:9 2026-02-13T02:08:04.684 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:08:04.684 INFO:tasks.workunit.client.0.trial112.stderr:6 2026-02-13T02:08:04.684 INFO:tasks.workunit.client.0.trial112.stderr:2 2026-02-13T02:08:04.685 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:08:04.685 INFO:tasks.workunit.client.0.trial112.stderr:7' 2026-02-13T02:08:04.685 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 9 0 6 2 1 7 2026-02-13T02:08:04.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: osds=('5' '9' '0' '6' '2' '1' '7') 2026-02-13T02:08:04.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: local -a osds 2026-02-13T02:08:04.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:315: corrupt_and_repair_erasure_coded: local not_primary_first=5 2026-02-13T02:08:04.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:316: corrupt_and_repair_erasure_coded: local not_primary_second=9 2026-02-13T02:08:04.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:319: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2026-02-13T02:08:04.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:286: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2026-02-13T02:08:04.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: corrupt_and_repair_one: local poolname=ecpool 2026-02-13T02:08:04.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:288: corrupt_and_repair_one: local osd=3 2026-02-13T02:08:04.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:293: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-13T02:08:04.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:08:04.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:08:04.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-13T02:08:04.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:08:04.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2026-02-13T02:08:04.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:08:04.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:08:04.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-13T02:08:04.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:08:04.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-13T02:08:04.689 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:08:04.689 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:08:04.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:08:04.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:08:04.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:08:04.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:08:04.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2026-02-13T02:08:04.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:08:04.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:08:04.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-13T02:08:04.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:08:04.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:08:04.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2026-02-13T02:08:05.177 INFO:tasks.workunit.client.0.trial112.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2026-02-13T02:08:05.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-13T02:08:05.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:08:05.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:08:05.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-13T02:08:05.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:08:05.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:08:05.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:08:05.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:08:05.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:08:05.289 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:08:05.289 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-13T02:08:05.289 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-13T02:08:05.289 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:08:05.289 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:08:05.289 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:08:05.289 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:08:05.289 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:08:05.289 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:08:05.289 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:08:05.289 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:08:05.289 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:08:05.289 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:08:05.291 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:08:05.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:08:05.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:08:05.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:08:05.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:08:05.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:08:05.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:08:05.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:08:05.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:08:05.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:08:05.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-13T02:08:05.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-13T02:08:05.293 INFO:tasks.workunit.client.0.trial112.stderr:start osd.3 2026-02-13T02:08:05.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:08:05.293 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-13T02:08:05.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-13T02:08:05.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:08:05.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:08:05.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:08:05.309 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:08:05.295+0000 7f94bb7bea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:08:05.310 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:08:05.299+0000 7f94bb7bea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:08:05.311 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:08:05.299+0000 7f94bb7bea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:08:05.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-13T02:08:05.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:08:05.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-13T02:08:05.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:08:05.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:08:05.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:08:05.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:08:05.462 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:08:05.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:08:05.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:08:05.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:08:05.691 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:08:05.679+0000 7f94bb7bea40 -1 Falling back to public interface 2026-02-13T02:08:06.337 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:08:06.323+0000 7f94bb7bea40 -1 osd.3 61 log_to_monitors true 2026-02-13T02:08:06.618 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:08:06.619 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:08:06.619 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:08:06.619 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:08:06.619 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:08:06.619 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:08:06.775 INFO:tasks.workunit.client.0.trial112.stderr:osd.3 up in weight 1 up_from 65 up_thru 65 down_at 62 last_clean_interval [18,61) [v2:127.0.0.1:6826/4053549015,v1:127.0.0.1:6827/4053549015] [v2:127.0.0.1:6828/4053549015,v1:127.0.0.1:6829/4053549015] exists,up eecbad03-21ed-4f9f-89e2-0664db8d736e 2026-02-13T02:08:06.775 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:08:06.775 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:08:06.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:08:06.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:08:06.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:08:06.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:08:06.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:08:06.776 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:08:06.777 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:08:06.777 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:08:06.777 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:08:06.777 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:08:06.777 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:08:06.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:08:06.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:08:06.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:08:06.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:08:06.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:08:06.890 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:08:07.050 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:08:07.050 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:08:07.050 INFO:tasks.workunit.client.0.trial112.stderr:2 2026-02-13T02:08:07.050 INFO:tasks.workunit.client.0.trial112.stderr:3 2026-02-13T02:08:07.050 INFO:tasks.workunit.client.0.trial112.stderr:4 2026-02-13T02:08:07.050 INFO:tasks.workunit.client.0.trial112.stderr:5 2026-02-13T02:08:07.050 INFO:tasks.workunit.client.0.trial112.stderr:6 2026-02-13T02:08:07.050 INFO:tasks.workunit.client.0.trial112.stderr:7 2026-02-13T02:08:07.050 INFO:tasks.workunit.client.0.trial112.stderr:8 2026-02-13T02:08:07.051 INFO:tasks.workunit.client.0.trial112.stderr:9' 2026-02-13T02:08:07.051 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:08:07.051 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:07.051 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:08:07.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836493 2026-02-13T02:08:07.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836493 2026-02-13T02:08:07.125 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493' 2026-02-13T02:08:07.125 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:07.125 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:08:07.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705677 2026-02-13T02:08:07.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705677 2026-02-13T02:08:07.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-38654705677' 2026-02-13T02:08:07.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:07.198 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:08:07.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542156 2026-02-13T02:08:07.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542156 2026-02-13T02:08:07.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-38654705677 2-60129542156' 2026-02-13T02:08:07.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:07.275 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:08:07.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=279172874242 2026-02-13T02:08:07.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 279172874242 2026-02-13T02:08:07.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-38654705677 2-60129542156 3-279172874242' 2026-02-13T02:08:07.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:07.353 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-13T02:08:07.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280522 2026-02-13T02:08:07.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280522 2026-02-13T02:08:07.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-38654705677 2-60129542156 3-279172874242 4-94489280522' 2026-02-13T02:08:07.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:07.427 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-13T02:08:07.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964117002 2026-02-13T02:08:07.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964117002 2026-02-13T02:08:07.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-38654705677 2-60129542156 3-279172874242 4-94489280522 5-115964117002' 2026-02-13T02:08:07.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:07.501 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-13T02:08:07.575 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986185 2026-02-13T02:08:07.575 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986185 2026-02-13T02:08:07.575 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-38654705677 2-60129542156 3-279172874242 4-94489280522 5-115964117002 6-133143986185' 2026-02-13T02:08:07.575 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:07.575 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-13T02:08:07.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855368 2026-02-13T02:08:07.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855368 2026-02-13T02:08:07.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-38654705677 2-60129542156 3-279172874242 4-94489280522 5-115964117002 6-133143986185 7-150323855368' 2026-02-13T02:08:07.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:07.651 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-13T02:08:07.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691848 2026-02-13T02:08:07.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691848 2026-02-13T02:08:07.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-38654705677 2-60129542156 3-279172874242 4-94489280522 5-115964117002 6-133143986185 7-150323855368 8-171798691848' 2026-02-13T02:08:07.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:07.725 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-13T02:08:07.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=193273528327 2026-02-13T02:08:07.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 193273528327 2026-02-13T02:08:07.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-38654705677 2-60129542156 3-279172874242 4-94489280522 5-115964117002 6-133143986185 7-150323855368 8-171798691848 9-193273528327' 2026-02-13T02:08:07.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:07.801 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836493 2026-02-13T02:08:07.801 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:07.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:08:07.803 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836493 2026-02-13T02:08:07.803 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:07.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836493 2026-02-13T02:08:07.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2026-02-13T02:08:07.804 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 21474836493 2026-02-13T02:08:07.805 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:08:07.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836492 -lt 21474836493 2026-02-13T02:08:07.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:08:08.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:08:08.980 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:08:09.138 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836494 -lt 21474836493 2026-02-13T02:08:09.138 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:09.139 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705677 2026-02-13T02:08:09.139 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:09.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:08:09.141 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705677 2026-02-13T02:08:09.141 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:09.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705677 2026-02-13T02:08:09.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705677' 2026-02-13T02:08:09.143 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 38654705677 2026-02-13T02:08:09.143 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:08:09.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705677 -lt 38654705677 2026-02-13T02:08:09.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:09.312 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542156 2026-02-13T02:08:09.312 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:09.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:08:09.314 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542156 2026-02-13T02:08:09.314 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:09.315 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542156 2026-02-13T02:08:09.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542156' 2026-02-13T02:08:09.316 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 60129542156 2026-02-13T02:08:09.316 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:08:09.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542156 -lt 60129542156 2026-02-13T02:08:09.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:09.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-279172874242 2026-02-13T02:08:09.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:09.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:08:09.490 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-279172874242 2026-02-13T02:08:09.490 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:09.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=279172874242 2026-02-13T02:08:09.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 279172874242' 2026-02-13T02:08:09.492 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.3 seq 279172874242 2026-02-13T02:08:09.492 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:08:09.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 279172874242 -lt 279172874242 2026-02-13T02:08:09.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:09.656 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-94489280522 2026-02-13T02:08:09.656 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:09.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-13T02:08:09.658 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-94489280522 2026-02-13T02:08:09.658 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:09.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280522 2026-02-13T02:08:09.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 94489280522' 2026-02-13T02:08:09.660 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.4 seq 94489280522 2026-02-13T02:08:09.660 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-13T02:08:09.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280523 -lt 94489280522 2026-02-13T02:08:09.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:09.834 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-115964117002 2026-02-13T02:08:09.834 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:09.836 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-13T02:08:09.836 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-115964117002 2026-02-13T02:08:09.837 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:09.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=115964117002 2026-02-13T02:08:09.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 115964117002' 2026-02-13T02:08:09.838 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.5 seq 115964117002 2026-02-13T02:08:09.839 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-13T02:08:10.001 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 115964117002 -lt 115964117002 2026-02-13T02:08:10.001 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:10.002 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-133143986185 2026-02-13T02:08:10.002 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:10.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-13T02:08:10.004 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-133143986185 2026-02-13T02:08:10.004 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:10.006 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=133143986185 2026-02-13T02:08:10.006 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 133143986185' 2026-02-13T02:08:10.006 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.6 seq 133143986185 2026-02-13T02:08:10.006 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-13T02:08:10.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 133143986185 -lt 133143986185 2026-02-13T02:08:10.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:10.170 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-150323855368 2026-02-13T02:08:10.170 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:10.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-13T02:08:10.172 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-150323855368 2026-02-13T02:08:10.172 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:10.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855368 2026-02-13T02:08:10.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 150323855368' 2026-02-13T02:08:10.174 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.7 seq 150323855368 2026-02-13T02:08:10.174 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-13T02:08:10.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855369 -lt 150323855368 2026-02-13T02:08:10.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:10.342 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-171798691848 2026-02-13T02:08:10.342 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:10.343 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-13T02:08:10.344 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-171798691848 2026-02-13T02:08:10.344 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:10.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691848 2026-02-13T02:08:10.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 171798691848' 2026-02-13T02:08:10.345 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.8 seq 171798691848 2026-02-13T02:08:10.345 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-13T02:08:10.516 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691848 -lt 171798691848 2026-02-13T02:08:10.516 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:10.517 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-193273528327 2026-02-13T02:08:10.517 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:10.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-13T02:08:10.519 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-193273528327 2026-02-13T02:08:10.520 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:10.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=193273528327 2026-02-13T02:08:10.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 193273528327' 2026-02-13T02:08:10.521 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.9 seq 193273528327 2026-02-13T02:08:10.522 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-13T02:08:10.678 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 193273528327 -lt 193273528327 2026-02-13T02:08:10.679 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:08:10.679 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:08:10.679 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:08:10.870 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:08:10.870 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:08:10.870 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:08:10.870 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:08:10.870 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:08:10.871 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:08:10.871 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:08:10.871 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:08:11.037 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:08:11.037 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:08:11.038 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:08:11.038 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:08:11.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:08:11.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:08:11.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:08:11.238 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: get_pg ecpool SOMETHING 2026-02-13T02:08:11.238 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-13T02:08:11.238 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-13T02:08:11.238 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-13T02:08:11.238 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T02:08:11.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: local pg=2.0 2026-02-13T02:08:11.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: corrupt_and_repair_one: repair 2.0 2026-02-13T02:08:11.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-13T02:08:11.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-13T02:08:11.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:08:11.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:08:11.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:08:11.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:08:11.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-13T02:07:58.240708+0000 2026-02-13T02:08:11.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-13T02:08:11.720 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-13T02:08:11.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-13T02:07:58.240708+0000 2026-02-13T02:08:11.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-13T02:08:11.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:07:58.240708+0000 2026-02-13T02:08:11.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:08:11.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:08:11.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:08:11.730 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:08:11.730 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:08:11.731 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:08:11.731 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:08:11.731 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:08:11.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:07:58.240708+0000 '>' 2026-02-13T02:07:58.240708+0000 2026-02-13T02:08:11.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:08:12.899 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:08:12.899 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:08:12.900 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:08:12.900 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:08:12.900 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:08:12.900 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:08:12.900 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:08:13.065 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:07:58.240708+0000 '>' 2026-02-13T02:07:58.240708+0000 2026-02-13T02:08:13.065 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:08:14.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:08:14.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:08:14.067 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:08:14.067 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:08:14.067 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:08:14.068 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:08:14.068 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:08:14.229 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:07:58.240708+0000 '>' 2026-02-13T02:07:58.240708+0000 2026-02-13T02:08:14.230 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:08:15.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:08:15.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:08:15.232 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:08:15.232 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:08:15.232 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:08:15.232 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:08:15.232 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:08:15.395 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:07:58.240708+0000 '>' 2026-02-13T02:07:58.240708+0000 2026-02-13T02:08:15.395 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:08:16.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:08:16.397 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:08:16.398 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:08:16.398 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:08:16.398 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:08:16.398 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:08:16.398 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:08:16.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:07:58.240708+0000 '>' 2026-02-13T02:07:58.240708+0000 2026-02-13T02:08:16.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:08:17.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:08:17.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:08:17.564 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:08:17.564 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:08:17.564 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:08:17.564 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:08:17.564 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:08:17.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:08:12.285519+0000 '>' 2026-02-13T02:07:58.240708+0000 2026-02-13T02:08:17.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:08:17.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-13T02:08:17.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:08:17.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:08:17.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2026-02-13T02:08:17.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:08:17.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-13T02:08:17.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:08:17.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:08:17.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2026-02-13T02:08:17.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:08:17.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2026-02-13T02:08:17.728 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:08:17.728 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:08:17.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:08:17.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:08:17.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:08:18.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:08:18.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-13T02:08:18.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:08:18.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:08:18.037 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2026-02-13T02:08:18.037 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:08:18.037 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2026-02-13T02:08:18.037 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2026-02-13T02:08:18.117 INFO:tasks.workunit.client.0.trial112.stdout:_ 2026-02-13T02:08:18.117 INFO:tasks.workunit.client.0.trial112.stdout:hinfo_key 2026-02-13T02:08:18.117 INFO:tasks.workunit.client.0.trial112.stdout:snapset 2026-02-13T02:08:18.219 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2026-02-13T02:08:18.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:08:18.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:08:18.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2026-02-13T02:08:18.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:08:18.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2026-02-13T02:08:18.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:08:18.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:08:18.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:08:18.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:08:18.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2026-02-13T02:08:18.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2026-02-13T02:08:18.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:08:18.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:08:18.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:08:18.221 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:08:18.221 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:08:18.221 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:08:18.221 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:08:18.221 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:08:18.221 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:08:18.221 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:08:18.223 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:08:18.223 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:08:18.224 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:08:18.224 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:08:18.224 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:08:18.224 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:08:18.224 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:08:18.224 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:08:18.224 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:08:18.224 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:08:18.224 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2026-02-13T02:08:18.224 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2026-02-13T02:08:18.224 INFO:tasks.workunit.client.0.trial112.stderr:start osd.9 2026-02-13T02:08:18.224 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:08:18.225 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2026-02-13T02:08:18.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2026-02-13T02:08:18.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:08:18.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:08:18.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:08:18.241 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:08:18.227+0000 7f5c714c9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:08:18.242 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:08:18.231+0000 7f5c714c9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:08:18.243 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:08:18.231+0000 7f5c714c9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:08:18.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2026-02-13T02:08:18.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:08:18.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2026-02-13T02:08:18.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:08:18.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:08:18.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:08:18.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:08:18.380 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:08:18.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:08:18.381 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-13T02:08:18.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:08:18.647 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:08:18.635+0000 7f5c714c9a40 -1 Falling back to public interface 2026-02-13T02:08:19.318 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:08:19.307+0000 7f5c714c9a40 -1 osd.9 68 log_to_monitors true 2026-02-13T02:08:19.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:08:19.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:08:19.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:08:19.545 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:08:19.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:08:19.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-13T02:08:19.709 INFO:tasks.workunit.client.0.trial112.stderr:osd.9 up in weight 1 up_from 71 up_thru 0 down_at 68 last_clean_interval [45,68) [v2:127.0.0.1:6874/2733401378,v1:127.0.0.1:6875/2733401378] [v2:127.0.0.1:6876/2733401378,v1:127.0.0.1:6877/2733401378] exists,up 18292385-2986-4859-a36e-1a5e00d7be2a 2026-02-13T02:08:19.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:08:19.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:08:19.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:08:19.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:08:19.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:08:19.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:08:19.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:08:19.710 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:08:19.711 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:08:19.711 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:08:19.711 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:08:19.711 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:08:19.711 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:08:19.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:08:19.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:08:19.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:08:19.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:08:19.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:08:19.826 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:08:19.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:08:19.982 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:08:19.982 INFO:tasks.workunit.client.0.trial112.stderr:2 2026-02-13T02:08:19.982 INFO:tasks.workunit.client.0.trial112.stderr:3 2026-02-13T02:08:19.982 INFO:tasks.workunit.client.0.trial112.stderr:4 2026-02-13T02:08:19.982 INFO:tasks.workunit.client.0.trial112.stderr:5 2026-02-13T02:08:19.982 INFO:tasks.workunit.client.0.trial112.stderr:6 2026-02-13T02:08:19.983 INFO:tasks.workunit.client.0.trial112.stderr:7 2026-02-13T02:08:19.983 INFO:tasks.workunit.client.0.trial112.stderr:8 2026-02-13T02:08:19.983 INFO:tasks.workunit.client.0.trial112.stderr:9' 2026-02-13T02:08:19.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:08:19.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:19.983 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:08:20.056 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836497 2026-02-13T02:08:20.056 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836497 2026-02-13T02:08:20.056 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497' 2026-02-13T02:08:20.056 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:20.056 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:08:20.128 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705680 2026-02-13T02:08:20.128 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705680 2026-02-13T02:08:20.128 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-38654705680' 2026-02-13T02:08:20.128 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:20.128 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:08:20.201 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542160 2026-02-13T02:08:20.201 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542160 2026-02-13T02:08:20.201 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-38654705680 2-60129542160' 2026-02-13T02:08:20.201 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:20.202 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:08:20.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=279172874245 2026-02-13T02:08:20.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 279172874245 2026-02-13T02:08:20.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-38654705680 2-60129542160 3-279172874245' 2026-02-13T02:08:20.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:20.276 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-13T02:08:20.347 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280526 2026-02-13T02:08:20.347 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280526 2026-02-13T02:08:20.347 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-38654705680 2-60129542160 3-279172874245 4-94489280526' 2026-02-13T02:08:20.347 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:20.347 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-13T02:08:20.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964117005 2026-02-13T02:08:20.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964117005 2026-02-13T02:08:20.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-38654705680 2-60129542160 3-279172874245 4-94489280526 5-115964117005' 2026-02-13T02:08:20.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:20.422 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-13T02:08:20.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986189 2026-02-13T02:08:20.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986189 2026-02-13T02:08:20.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-38654705680 2-60129542160 3-279172874245 4-94489280526 5-115964117005 6-133143986189' 2026-02-13T02:08:20.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:20.493 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-13T02:08:20.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855372 2026-02-13T02:08:20.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855372 2026-02-13T02:08:20.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-38654705680 2-60129542160 3-279172874245 4-94489280526 5-115964117005 6-133143986189 7-150323855372' 2026-02-13T02:08:20.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:20.566 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-13T02:08:20.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691851 2026-02-13T02:08:20.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691851 2026-02-13T02:08:20.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-38654705680 2-60129542160 3-279172874245 4-94489280526 5-115964117005 6-133143986189 7-150323855372 8-171798691851' 2026-02-13T02:08:20.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:20.638 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-13T02:08:20.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=304942678018 2026-02-13T02:08:20.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 304942678018 2026-02-13T02:08:20.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-38654705680 2-60129542160 3-279172874245 4-94489280526 5-115964117005 6-133143986189 7-150323855372 8-171798691851 9-304942678018' 2026-02-13T02:08:20.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:20.712 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836497 2026-02-13T02:08:20.712 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:20.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:08:20.714 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836497 2026-02-13T02:08:20.714 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:20.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836497 2026-02-13T02:08:20.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2026-02-13T02:08:20.716 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 21474836497 2026-02-13T02:08:20.716 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:08:20.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836497 -lt 21474836497 2026-02-13T02:08:20.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:20.881 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705680 2026-02-13T02:08:20.882 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:20.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:08:20.883 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705680 2026-02-13T02:08:20.884 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:20.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705680 2026-02-13T02:08:20.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705680' 2026-02-13T02:08:20.886 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 38654705680 2026-02-13T02:08:20.886 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:08:21.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705680 -lt 38654705680 2026-02-13T02:08:21.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:21.059 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542160 2026-02-13T02:08:21.059 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:21.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:08:21.061 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542160 2026-02-13T02:08:21.062 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:21.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542160 2026-02-13T02:08:21.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542160' 2026-02-13T02:08:21.063 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 60129542160 2026-02-13T02:08:21.064 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:08:21.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542160 -lt 60129542160 2026-02-13T02:08:21.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:21.239 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-279172874245 2026-02-13T02:08:21.240 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:21.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:08:21.242 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-279172874245 2026-02-13T02:08:21.242 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:21.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=279172874245 2026-02-13T02:08:21.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 279172874245' 2026-02-13T02:08:21.244 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.3 seq 279172874245 2026-02-13T02:08:21.244 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:08:21.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 279172874245 -lt 279172874245 2026-02-13T02:08:21.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:21.406 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-94489280526 2026-02-13T02:08:21.406 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:21.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-13T02:08:21.408 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-94489280526 2026-02-13T02:08:21.408 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:21.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280526 2026-02-13T02:08:21.410 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 94489280526' 2026-02-13T02:08:21.410 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.4 seq 94489280526 2026-02-13T02:08:21.410 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-13T02:08:21.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280526 -lt 94489280526 2026-02-13T02:08:21.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:21.583 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-115964117005 2026-02-13T02:08:21.583 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:21.584 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-13T02:08:21.585 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-115964117005 2026-02-13T02:08:21.585 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:21.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=115964117005 2026-02-13T02:08:21.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 115964117005' 2026-02-13T02:08:21.586 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.5 seq 115964117005 2026-02-13T02:08:21.587 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-13T02:08:21.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 115964117005 -lt 115964117005 2026-02-13T02:08:21.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:21.753 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-133143986189 2026-02-13T02:08:21.753 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:21.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-13T02:08:21.755 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-133143986189 2026-02-13T02:08:21.755 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:21.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=133143986189 2026-02-13T02:08:21.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 133143986189' 2026-02-13T02:08:21.757 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.6 seq 133143986189 2026-02-13T02:08:21.758 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-13T02:08:21.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 133143986189 -lt 133143986189 2026-02-13T02:08:21.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:21.932 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-150323855372 2026-02-13T02:08:21.932 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:21.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-13T02:08:21.934 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-150323855372 2026-02-13T02:08:21.934 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:21.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855372 2026-02-13T02:08:21.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 150323855372' 2026-02-13T02:08:21.936 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.7 seq 150323855372 2026-02-13T02:08:21.936 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-13T02:08:22.108 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855372 -lt 150323855372 2026-02-13T02:08:22.108 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:22.109 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-171798691851 2026-02-13T02:08:22.109 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:22.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-13T02:08:22.111 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-171798691851 2026-02-13T02:08:22.111 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:22.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691851 2026-02-13T02:08:22.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 171798691851' 2026-02-13T02:08:22.113 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.8 seq 171798691851 2026-02-13T02:08:22.113 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-13T02:08:22.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691851 -lt 171798691851 2026-02-13T02:08:22.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:22.285 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-304942678018 2026-02-13T02:08:22.286 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:22.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-13T02:08:22.288 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-304942678018 2026-02-13T02:08:22.288 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:22.289 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=304942678018 2026-02-13T02:08:22.289 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 304942678018' 2026-02-13T02:08:22.290 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.9 seq 304942678018 2026-02-13T02:08:22.290 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-13T02:08:22.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 304942678018 -lt 304942678018 2026-02-13T02:08:22.463 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:08:22.464 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:08:22.464 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:08:22.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:08:22.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:08:22.658 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:08:22.658 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:08:22.659 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:08:22.659 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:08:22.659 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:08:22.659 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:08:22.816 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:08:22.817 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:08:22.817 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:08:22.817 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:08:23.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:08:23.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:08:23.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:08:23.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-13T02:08:23.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-13T02:08:23.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:321: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2026-02-13T02:08:23.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:286: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2026-02-13T02:08:23.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: corrupt_and_repair_one: local poolname=ecpool 2026-02-13T02:08:23.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:288: corrupt_and_repair_one: local osd=5 2026-02-13T02:08:23.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:293: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2026-02-13T02:08:23.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:08:23.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:08:23.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2026-02-13T02:08:23.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:08:23.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2026-02-13T02:08:23.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:08:23.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:08:23.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2026-02-13T02:08:23.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:08:23.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2026-02-13T02:08:23.035 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:08:23.035 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:08:23.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:08:23.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:08:23.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:08:23.343 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:08:23.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2026-02-13T02:08:23.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:08:23.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:08:23.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2026-02-13T02:08:23.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:08:23.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2026-02-13T02:08:23.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2026-02-13T02:08:23.538 INFO:tasks.workunit.client.0.trial112.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2026-02-13T02:08:23.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2026-02-13T02:08:23.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:08:23.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:08:23.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2026-02-13T02:08:23.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:08:23.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2026-02-13T02:08:23.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:08:23.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:08:23.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:08:23.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:08:23.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2026-02-13T02:08:23.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2026-02-13T02:08:23.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:08:23.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:08:23.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:08:23.630 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:08:23.630 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:08:23.630 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:08:23.630 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:08:23.631 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:08:23.631 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:08:23.631 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:08:23.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:08:23.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:08:23.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:08:23.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:08:23.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:08:23.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:08:23.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:08:23.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:08:23.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:08:23.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:08:23.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2026-02-13T02:08:23.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2026-02-13T02:08:23.633 INFO:tasks.workunit.client.0.trial112.stderr:start osd.5 2026-02-13T02:08:23.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:08:23.633 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2026-02-13T02:08:23.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2026-02-13T02:08:23.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:08:23.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:08:23.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:08:23.646 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:08:23.634+0000 7f9116f40a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:08:23.647 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:08:23.634+0000 7f9116f40a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:08:23.648 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:08:23.634+0000 7f9116f40a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:08:23.795 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2026-02-13T02:08:23.795 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:08:23.795 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-13T02:08:23.795 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:08:23.795 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:08:23.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:08:23.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:08:23.796 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:08:23.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:08:23.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-13T02:08:23.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:08:24.055 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:08:24.042+0000 7f9116f40a40 -1 Falling back to public interface 2026-02-13T02:08:24.665 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:08:24.650+0000 7f9116f40a40 -1 osd.5 73 log_to_monitors true 2026-02-13T02:08:24.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:08:24.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:08:24.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:08:24.962 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:08:24.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:08:24.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-13T02:08:25.120 INFO:tasks.workunit.client.0.trial112.stderr:osd.5 up in weight 1 up_from 77 up_thru 77 down_at 74 last_clean_interval [27,73) [v2:127.0.0.1:6842/3959584031,v1:127.0.0.1:6843/3959584031] [v2:127.0.0.1:6844/3959584031,v1:127.0.0.1:6845/3959584031] exists,up 03eef3ce-a0e3-4fec-86e3-c40fb660505f 2026-02-13T02:08:25.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:08:25.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:08:25.121 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:08:25.121 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:08:25.121 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:08:25.121 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:08:25.121 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:08:25.121 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:08:25.122 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:08:25.122 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:08:25.122 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:08:25.122 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:08:25.122 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:08:25.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:08:25.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:08:25.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:08:25.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:08:25.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:08:25.235 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:08:25.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:08:25.392 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:08:25.392 INFO:tasks.workunit.client.0.trial112.stderr:2 2026-02-13T02:08:25.392 INFO:tasks.workunit.client.0.trial112.stderr:3 2026-02-13T02:08:25.392 INFO:tasks.workunit.client.0.trial112.stderr:4 2026-02-13T02:08:25.392 INFO:tasks.workunit.client.0.trial112.stderr:5 2026-02-13T02:08:25.392 INFO:tasks.workunit.client.0.trial112.stderr:6 2026-02-13T02:08:25.392 INFO:tasks.workunit.client.0.trial112.stderr:7 2026-02-13T02:08:25.392 INFO:tasks.workunit.client.0.trial112.stderr:8 2026-02-13T02:08:25.392 INFO:tasks.workunit.client.0.trial112.stderr:9' 2026-02-13T02:08:25.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:08:25.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:25.392 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:08:25.465 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836499 2026-02-13T02:08:25.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836499 2026-02-13T02:08:25.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499' 2026-02-13T02:08:25.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:25.466 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:08:25.536 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705682 2026-02-13T02:08:25.536 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705682 2026-02-13T02:08:25.536 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-38654705682' 2026-02-13T02:08:25.536 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:25.537 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:08:25.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542162 2026-02-13T02:08:25.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542162 2026-02-13T02:08:25.610 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-38654705682 2-60129542162' 2026-02-13T02:08:25.610 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:25.610 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:08:25.681 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=279172874247 2026-02-13T02:08:25.681 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 279172874247 2026-02-13T02:08:25.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-38654705682 2-60129542162 3-279172874247' 2026-02-13T02:08:25.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:25.682 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-13T02:08:25.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280528 2026-02-13T02:08:25.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280528 2026-02-13T02:08:25.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-38654705682 2-60129542162 3-279172874247 4-94489280528' 2026-02-13T02:08:25.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:25.757 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-13T02:08:25.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=330712481794 2026-02-13T02:08:25.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 330712481794 2026-02-13T02:08:25.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-38654705682 2-60129542162 3-279172874247 4-94489280528 5-330712481794' 2026-02-13T02:08:25.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:25.831 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-13T02:08:25.905 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986191 2026-02-13T02:08:25.905 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986191 2026-02-13T02:08:25.905 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-38654705682 2-60129542162 3-279172874247 4-94489280528 5-330712481794 6-133143986191' 2026-02-13T02:08:25.905 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:25.906 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-13T02:08:25.980 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855374 2026-02-13T02:08:25.980 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855374 2026-02-13T02:08:25.980 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-38654705682 2-60129542162 3-279172874247 4-94489280528 5-330712481794 6-133143986191 7-150323855374' 2026-02-13T02:08:25.980 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:25.980 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-13T02:08:26.051 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691853 2026-02-13T02:08:26.051 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691853 2026-02-13T02:08:26.051 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-38654705682 2-60129542162 3-279172874247 4-94489280528 5-330712481794 6-133143986191 7-150323855374 8-171798691853' 2026-02-13T02:08:26.051 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:26.051 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-13T02:08:26.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=304942678020 2026-02-13T02:08:26.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 304942678020 2026-02-13T02:08:26.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-38654705682 2-60129542162 3-279172874247 4-94489280528 5-330712481794 6-133143986191 7-150323855374 8-171798691853 9-304942678020' 2026-02-13T02:08:26.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:26.128 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836499 2026-02-13T02:08:26.128 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:26.129 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:08:26.130 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836499 2026-02-13T02:08:26.130 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:26.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836499 2026-02-13T02:08:26.132 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2026-02-13T02:08:26.132 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 21474836499 2026-02-13T02:08:26.132 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:08:26.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836498 -lt 21474836499 2026-02-13T02:08:26.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:08:27.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:08:27.295 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:08:27.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836499 -lt 21474836499 2026-02-13T02:08:27.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:27.460 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705682 2026-02-13T02:08:27.461 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:27.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:08:27.463 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705682 2026-02-13T02:08:27.463 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:27.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705682 2026-02-13T02:08:27.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705682' 2026-02-13T02:08:27.465 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 38654705682 2026-02-13T02:08:27.465 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:08:27.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705683 -lt 38654705682 2026-02-13T02:08:27.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:27.633 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542162 2026-02-13T02:08:27.633 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:27.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:08:27.634 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542162 2026-02-13T02:08:27.635 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:27.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542162 2026-02-13T02:08:27.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542162' 2026-02-13T02:08:27.636 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 60129542162 2026-02-13T02:08:27.637 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:08:27.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542162 -lt 60129542162 2026-02-13T02:08:27.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:27.810 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-279172874247 2026-02-13T02:08:27.810 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:27.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:08:27.812 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-279172874247 2026-02-13T02:08:27.812 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:27.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=279172874247 2026-02-13T02:08:27.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 279172874247' 2026-02-13T02:08:27.814 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.3 seq 279172874247 2026-02-13T02:08:27.814 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:08:27.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 279172874248 -lt 279172874247 2026-02-13T02:08:27.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:27.983 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-94489280528 2026-02-13T02:08:27.983 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:27.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-13T02:08:27.985 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-94489280528 2026-02-13T02:08:27.985 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:27.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280528 2026-02-13T02:08:27.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 94489280528' 2026-02-13T02:08:27.987 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.4 seq 94489280528 2026-02-13T02:08:27.987 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-13T02:08:28.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280528 -lt 94489280528 2026-02-13T02:08:28.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:28.158 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-330712481794 2026-02-13T02:08:28.158 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:28.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-13T02:08:28.160 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-330712481794 2026-02-13T02:08:28.160 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:28.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=330712481794 2026-02-13T02:08:28.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 330712481794' 2026-02-13T02:08:28.162 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.5 seq 330712481794 2026-02-13T02:08:28.163 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-13T02:08:28.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 330712481794 -lt 330712481794 2026-02-13T02:08:28.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:28.324 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-133143986191 2026-02-13T02:08:28.324 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:28.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-13T02:08:28.326 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-133143986191 2026-02-13T02:08:28.327 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:28.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=133143986191 2026-02-13T02:08:28.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 133143986191' 2026-02-13T02:08:28.328 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.6 seq 133143986191 2026-02-13T02:08:28.329 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-13T02:08:28.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 133143986191 -lt 133143986191 2026-02-13T02:08:28.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:28.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-150323855374 2026-02-13T02:08:28.489 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:28.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-13T02:08:28.490 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-150323855374 2026-02-13T02:08:28.491 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:28.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855374 2026-02-13T02:08:28.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 150323855374' 2026-02-13T02:08:28.492 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.7 seq 150323855374 2026-02-13T02:08:28.493 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-13T02:08:28.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855374 -lt 150323855374 2026-02-13T02:08:28.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:28.667 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-171798691853 2026-02-13T02:08:28.667 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:28.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-13T02:08:28.669 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-171798691853 2026-02-13T02:08:28.669 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:28.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691853 2026-02-13T02:08:28.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 171798691853' 2026-02-13T02:08:28.671 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.8 seq 171798691853 2026-02-13T02:08:28.671 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-13T02:08:28.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691854 -lt 171798691853 2026-02-13T02:08:28.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:28.829 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-304942678020 2026-02-13T02:08:28.830 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:28.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-13T02:08:28.831 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-304942678020 2026-02-13T02:08:28.832 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:28.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=304942678020 2026-02-13T02:08:28.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 304942678020' 2026-02-13T02:08:28.833 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.9 seq 304942678020 2026-02-13T02:08:28.834 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-13T02:08:29.006 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 304942678020 -lt 304942678020 2026-02-13T02:08:29.006 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:08:29.007 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:08:29.007 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:08:29.206 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:08:29.206 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:08:29.207 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:08:29.207 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:08:29.207 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:08:29.207 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:08:29.207 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:08:29.208 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:08:29.368 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:08:29.369 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:08:29.369 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:08:29.369 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:08:29.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:08:29.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:08:29.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:08:29.567 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: get_pg ecpool SOMETHING 2026-02-13T02:08:29.567 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-13T02:08:29.567 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-13T02:08:29.567 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-13T02:08:29.568 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T02:08:29.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: local pg=2.0 2026-02-13T02:08:29.729 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: corrupt_and_repair_one: repair 2.0 2026-02-13T02:08:29.729 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-13T02:08:29.729 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-13T02:08:29.729 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:08:29.729 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:08:29.730 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:08:29.730 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:08:29.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-13T02:08:12.285519+0000 2026-02-13T02:08:29.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-13T02:08:30.038 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-13T02:08:30.049 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-13T02:08:12.285519+0000 2026-02-13T02:08:30.049 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-13T02:08:30.049 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:08:12.285519+0000 2026-02-13T02:08:30.049 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:08:30.049 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:08:30.049 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:08:30.049 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:08:30.049 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:08:30.049 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:08:30.050 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:08:30.050 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:08:30.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:08:12.285519+0000 '>' 2026-02-13T02:08:12.285519+0000 2026-02-13T02:08:30.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:08:31.222 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:08:31.222 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:08:31.223 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:08:31.223 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:08:31.223 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:08:31.223 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:08:31.224 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:08:31.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:08:12.285519+0000 '>' 2026-02-13T02:08:12.285519+0000 2026-02-13T02:08:31.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:08:32.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:08:32.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:08:32.389 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:08:32.389 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:08:32.389 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:08:32.389 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:08:32.389 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:08:32.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:08:12.285519+0000 '>' 2026-02-13T02:08:12.285519+0000 2026-02-13T02:08:32.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:08:33.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:08:33.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:08:33.554 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:08:33.554 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:08:33.554 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:08:33.555 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:08:33.555 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:08:33.720 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:08:30.289141+0000 '>' 2026-02-13T02:08:12.285519+0000 2026-02-13T02:08:33.720 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:08:33.720 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-13T02:08:33.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:08:33.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:08:33.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2026-02-13T02:08:33.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:08:33.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-13T02:08:33.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:08:33.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:08:33.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2026-02-13T02:08:33.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:08:33.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2026-02-13T02:08:33.721 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:08:33.721 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:08:33.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:08:33.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:08:33.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:08:34.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:08:34.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-13T02:08:34.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:08:34.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:08:34.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2026-02-13T02:08:34.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:08:34.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2026-02-13T02:08:34.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2026-02-13T02:08:34.105 INFO:tasks.workunit.client.0.trial112.stdout:_ 2026-02-13T02:08:34.105 INFO:tasks.workunit.client.0.trial112.stdout:hinfo_key 2026-02-13T02:08:34.105 INFO:tasks.workunit.client.0.trial112.stdout:snapset 2026-02-13T02:08:34.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2026-02-13T02:08:34.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:08:34.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:08:34.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2026-02-13T02:08:34.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:08:34.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2026-02-13T02:08:34.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:08:34.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:08:34.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:08:34.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:08:34.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2026-02-13T02:08:34.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2026-02-13T02:08:34.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:08:34.217 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:08:34.217 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:08:34.217 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:08:34.217 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:08:34.217 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:08:34.217 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:08:34.217 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:08:34.217 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:08:34.217 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:08:34.219 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:08:34.219 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:08:34.219 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:08:34.219 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:08:34.219 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:08:34.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:08:34.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:08:34.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:08:34.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:08:34.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:08:34.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2026-02-13T02:08:34.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2026-02-13T02:08:34.220 INFO:tasks.workunit.client.0.trial112.stderr:start osd.9 2026-02-13T02:08:34.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:08:34.220 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2026-02-13T02:08:34.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2026-02-13T02:08:34.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:08:34.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:08:34.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:08:34.240 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:08:34.230+0000 7f823d4b9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:08:34.241 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:08:34.230+0000 7f823d4b9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:08:34.242 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:08:34.234+0000 7f823d4b9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:08:34.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2026-02-13T02:08:34.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:08:34.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2026-02-13T02:08:34.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:08:34.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:08:34.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:08:34.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:08:34.380 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:08:34.381 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:08:34.381 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-13T02:08:34.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:08:34.667 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:08:34.658+0000 7f823d4b9a40 -1 Falling back to public interface 2026-02-13T02:08:35.297 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:08:35.286+0000 7f823d4b9a40 -1 osd.9 79 log_to_monitors true 2026-02-13T02:08:35.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:08:35.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:08:35.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:08:35.544 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:08:35.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:08:35.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-13T02:08:35.700 INFO:tasks.workunit.client.0.trial112.stderr:osd.9 up in weight 1 up_from 83 up_thru 0 down_at 80 last_clean_interval [71,79) [v2:127.0.0.1:6874/1642458599,v1:127.0.0.1:6875/1642458599] [v2:127.0.0.1:6876/1642458599,v1:127.0.0.1:6877/1642458599] exists,up 18292385-2986-4859-a36e-1a5e00d7be2a 2026-02-13T02:08:35.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:08:35.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:08:35.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:08:35.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:08:35.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:08:35.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:08:35.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:08:35.701 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:08:35.702 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:08:35.702 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:08:35.702 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:08:35.702 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:08:35.703 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:08:35.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:08:35.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:08:35.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:08:35.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:08:35.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:08:35.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:08:35.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:08:35.969 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:08:35.970 INFO:tasks.workunit.client.0.trial112.stderr:2 2026-02-13T02:08:35.970 INFO:tasks.workunit.client.0.trial112.stderr:3 2026-02-13T02:08:35.970 INFO:tasks.workunit.client.0.trial112.stderr:4 2026-02-13T02:08:35.970 INFO:tasks.workunit.client.0.trial112.stderr:5 2026-02-13T02:08:35.970 INFO:tasks.workunit.client.0.trial112.stderr:6 2026-02-13T02:08:35.970 INFO:tasks.workunit.client.0.trial112.stderr:7 2026-02-13T02:08:35.970 INFO:tasks.workunit.client.0.trial112.stderr:8 2026-02-13T02:08:35.970 INFO:tasks.workunit.client.0.trial112.stderr:9' 2026-02-13T02:08:35.970 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:08:35.970 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:35.970 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:08:36.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836502 2026-02-13T02:08:36.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836502 2026-02-13T02:08:36.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502' 2026-02-13T02:08:36.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:36.041 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:08:36.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705685 2026-02-13T02:08:36.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705685 2026-02-13T02:08:36.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-38654705685' 2026-02-13T02:08:36.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:36.115 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:08:36.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542165 2026-02-13T02:08:36.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542165 2026-02-13T02:08:36.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-38654705685 2-60129542165' 2026-02-13T02:08:36.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:36.187 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:08:36.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=279172874250 2026-02-13T02:08:36.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 279172874250 2026-02-13T02:08:36.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-38654705685 2-60129542165 3-279172874250' 2026-02-13T02:08:36.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:36.261 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-13T02:08:36.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280531 2026-02-13T02:08:36.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280531 2026-02-13T02:08:36.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-38654705685 2-60129542165 3-279172874250 4-94489280531' 2026-02-13T02:08:36.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:36.334 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-13T02:08:36.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=330712481797 2026-02-13T02:08:36.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 330712481797 2026-02-13T02:08:36.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-38654705685 2-60129542165 3-279172874250 4-94489280531 5-330712481797' 2026-02-13T02:08:36.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:36.409 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-13T02:08:36.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986194 2026-02-13T02:08:36.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986194 2026-02-13T02:08:36.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-38654705685 2-60129542165 3-279172874250 4-94489280531 5-330712481797 6-133143986194' 2026-02-13T02:08:36.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:36.482 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-13T02:08:36.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855377 2026-02-13T02:08:36.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855377 2026-02-13T02:08:36.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-38654705685 2-60129542165 3-279172874250 4-94489280531 5-330712481797 6-133143986194 7-150323855377' 2026-02-13T02:08:36.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:36.553 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-13T02:08:36.626 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691856 2026-02-13T02:08:36.626 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691856 2026-02-13T02:08:36.626 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-38654705685 2-60129542165 3-279172874250 4-94489280531 5-330712481797 6-133143986194 7-150323855377 8-171798691856' 2026-02-13T02:08:36.626 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:36.626 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-13T02:08:36.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=356482285570 2026-02-13T02:08:36.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 356482285570 2026-02-13T02:08:36.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-38654705685 2-60129542165 3-279172874250 4-94489280531 5-330712481797 6-133143986194 7-150323855377 8-171798691856 9-356482285570' 2026-02-13T02:08:36.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:36.697 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836502 2026-02-13T02:08:36.698 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:36.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:08:36.699 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836502 2026-02-13T02:08:36.699 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:36.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836502 2026-02-13T02:08:36.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2026-02-13T02:08:36.701 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 21474836502 2026-02-13T02:08:36.701 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:08:36.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836502 -lt 21474836502 2026-02-13T02:08:36.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:36.866 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705685 2026-02-13T02:08:36.866 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:36.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:08:36.868 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705685 2026-02-13T02:08:36.868 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:36.870 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705685 2026-02-13T02:08:36.870 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705685' 2026-02-13T02:08:36.870 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 38654705685 2026-02-13T02:08:36.870 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:08:37.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705686 -lt 38654705685 2026-02-13T02:08:37.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:37.040 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542165 2026-02-13T02:08:37.040 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:37.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:08:37.042 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542165 2026-02-13T02:08:37.042 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:37.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542165 2026-02-13T02:08:37.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542165' 2026-02-13T02:08:37.044 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 60129542165 2026-02-13T02:08:37.044 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:08:37.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542165 -lt 60129542165 2026-02-13T02:08:37.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:37.213 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-279172874250 2026-02-13T02:08:37.213 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:37.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:08:37.216 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-279172874250 2026-02-13T02:08:37.216 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:37.217 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=279172874250 2026-02-13T02:08:37.217 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 279172874250' 2026-02-13T02:08:37.217 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.3 seq 279172874250 2026-02-13T02:08:37.218 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:08:37.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 279172874251 -lt 279172874250 2026-02-13T02:08:37.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:37.392 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-94489280531 2026-02-13T02:08:37.392 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:37.393 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-13T02:08:37.394 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-94489280531 2026-02-13T02:08:37.394 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:37.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280531 2026-02-13T02:08:37.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 94489280531' 2026-02-13T02:08:37.396 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.4 seq 94489280531 2026-02-13T02:08:37.396 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-13T02:08:37.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280531 -lt 94489280531 2026-02-13T02:08:37.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:37.555 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-330712481797 2026-02-13T02:08:37.555 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:37.557 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-13T02:08:37.557 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-330712481797 2026-02-13T02:08:37.558 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:37.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=330712481797 2026-02-13T02:08:37.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 330712481797' 2026-02-13T02:08:37.560 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.5 seq 330712481797 2026-02-13T02:08:37.560 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-13T02:08:37.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 330712481797 -lt 330712481797 2026-02-13T02:08:37.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:37.734 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-133143986194 2026-02-13T02:08:37.735 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:37.735 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-13T02:08:37.736 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-133143986194 2026-02-13T02:08:37.736 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:37.737 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=133143986194 2026-02-13T02:08:37.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 133143986194' 2026-02-13T02:08:37.738 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.6 seq 133143986194 2026-02-13T02:08:37.738 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-13T02:08:37.908 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 133143986194 -lt 133143986194 2026-02-13T02:08:37.908 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:37.909 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-150323855377 2026-02-13T02:08:37.909 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:37.910 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-13T02:08:37.911 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-150323855377 2026-02-13T02:08:37.911 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:37.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855377 2026-02-13T02:08:37.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 150323855377' 2026-02-13T02:08:37.912 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.7 seq 150323855377 2026-02-13T02:08:37.913 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-13T02:08:38.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855377 -lt 150323855377 2026-02-13T02:08:38.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:38.076 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-171798691856 2026-02-13T02:08:38.076 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:38.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-13T02:08:38.079 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-171798691856 2026-02-13T02:08:38.079 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:38.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691856 2026-02-13T02:08:38.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 171798691856' 2026-02-13T02:08:38.080 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.8 seq 171798691856 2026-02-13T02:08:38.080 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-13T02:08:38.253 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691856 -lt 171798691856 2026-02-13T02:08:38.253 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:38.254 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-356482285570 2026-02-13T02:08:38.254 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:38.255 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-13T02:08:38.256 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-356482285570 2026-02-13T02:08:38.256 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:38.258 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=356482285570 2026-02-13T02:08:38.258 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 356482285570' 2026-02-13T02:08:38.258 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.9 seq 356482285570 2026-02-13T02:08:38.259 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-13T02:08:38.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 356482285570 -lt 356482285570 2026-02-13T02:08:38.422 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:08:38.423 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:08:38.423 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:08:38.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:08:38.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:08:38.615 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:08:38.615 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:08:38.615 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:08:38.615 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:08:38.615 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:08:38.615 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:08:38.770 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-13T02:08:38.771 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:08:38.771 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:08:38.771 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:08:38.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-13T02:08:38.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' -1 2026-02-13T02:08:38.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-13T02:08:38.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=4 2026-02-13T02:08:38.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-13T02:08:38.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-13T02:08:39.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-13T02:08:39.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:08:39.064 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:08:39.064 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:08:39.064 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:08:39.064 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:08:39.064 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:08:39.064 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:08:39.219 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:08:39.220 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:08:39.220 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:08:39.220 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:08:39.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:08:39.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:08:39.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:08:39.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-13T02:08:39.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-13T02:08:39.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:322: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2026-02-13T02:08:39.435 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:245: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2026-02-13T02:08:39.435 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:246: corrupt_and_repair_two: local poolname=ecpool 2026-02-13T02:08:39.435 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:247: corrupt_and_repair_two: local first=5 2026-02-13T02:08:39.435 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_two: local second=9 2026-02-13T02:08:39.435 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_two: pids= 2026-02-13T02:08:39.435 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2026-02-13T02:08:39.435 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-13T02:08:39.435 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-13T02:08:39.435 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 218466"' 2026-02-13T02:08:39.435 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 218466' 2026-02-13T02:08:39.435 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2026-02-13T02:08:39.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-13T02:08:39.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-13T02:08:39.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 218468"' 2026-02-13T02:08:39.436 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 218468' 2026-02-13T02:08:39.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2026-02-13T02:08:39.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_two: wait_background pids 2026-02-13T02:08:39.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 218466 218468' 2026-02-13T02:08:39.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-13T02:08:39.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-13T02:08:39.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 218466 2026-02-13T02:08:39.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/218469: /' 2026-02-13T02:08:39.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2026-02-13T02:08:39.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/218471: /' 2026-02-13T02:08:40.050 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:08:40.050 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:08:40.050 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2026-02-13T02:08:40.051 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:08:40.051 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2026-02-13T02:08:40.051 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:08:40.051 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:08:40.051 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2026-02-13T02:08:40.051 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:08:40.051 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2026-02-13T02:08:40.051 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:08:40.051 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:08:40.051 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:08:40.051 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:08:40.051 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:08:40.052 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:08:40.052 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2026-02-13T02:08:40.052 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:08:40.052 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:08:40.052 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2026-02-13T02:08:40.052 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:08:40.052 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2026-02-13T02:08:40.052 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2026-02-13T02:08:40.052 INFO:tasks.workunit.client.0.trial112.stderr:218469: remove 1#2:eb822e21:::SOMETHING:head# 2026-02-13T02:08:40.052 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2026-02-13T02:08:40.052 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:08:40.052 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:08:40.053 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2026-02-13T02:08:40.053 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:08:40.053 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2026-02-13T02:08:40.053 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:08:40.053 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:08:40.054 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:08:40.054 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:08:40.054 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2026-02-13T02:08:40.055 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2026-02-13T02:08:40.055 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:08:40.055 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:08:40.055 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:08:40.055 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:08:40.055 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:08:40.055 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:08:40.055 INFO:tasks.workunit.client.0.trial112.stderr:218469: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:08:40.055 INFO:tasks.workunit.client.0.trial112.stderr:218469: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:08:40.055 INFO:tasks.workunit.client.0.trial112.stderr:218469: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:08:40.055 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:08:40.055 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:08:40.056 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:08:40.056 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:08:40.056 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:08:40.056 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:08:40.056 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:08:40.056 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:08:40.056 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:08:40.056 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:08:40.056 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:08:40.056 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2026-02-13T02:08:40.056 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2026-02-13T02:08:40.056 INFO:tasks.workunit.client.0.trial112.stderr:218469: start osd.5 2026-02-13T02:08:40.056 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:08:40.058 INFO:tasks.workunit.client.0.trial112.stderr:2218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:08:40.058 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:08:40.058 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2026-02-13T02:08:40.059 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:08:40.059 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2026-02-13T02:08:40.059 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:08:40.059 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:08:40.059 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2026-02-13T02:08:40.059 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:08:40.059 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2026-02-13T02:08:40.059 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:08:40.059 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:08:40.059 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:08:40.059 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:08:40.059 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:08:40.060 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:08:40.060 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2026-02-13T02:08:40.060 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:08:40.060 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:08:40.060 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2026-02-13T02:08:40.060 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:08:40.060 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2026-02-13T02:08:40.060 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2026-02-13T02:08:40.060 INFO:tasks.workunit.client.0.trial112.stderr:218471: remove 2#2:eb822e21:::SOMETHING:head# 2026-02-13T02:08:40.060 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2026-02-13T02:08:40.060 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:08:40.060 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:08:40.061 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2026-02-13T02:08:40.061 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:08:40.061 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2026-02-13T02:08:40.061 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:08:40.061 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:08:40.061 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:08:40.062 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:08:40.062 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2026-02-13T02:08:40.062 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2026-02-13T02:08:40.062 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:08:40.062 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:08:40.062 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:08:40.062 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:08:40.062 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:08:40.062 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:08:40.062 INFO:tasks.workunit.client.0.trial112.stderr:218471: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:08:40.062 INFO:tasks.workunit.client.0.trial112.stderr:218471: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:08:40.063 INFO:tasks.workunit.client.0.trial112.stderr:218471: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:08:40.063 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:08:40.063 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:08:40.063 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:08:40.063 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:08:40.063 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:08:40.063 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:08:40.063 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:08:40.063 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:08:40.063 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:08:40.063 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:08:40.063 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:08:40.063 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2026-02-13T02:08:40.064 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2026-02-13T02:08:40.064 INFO:tasks.workunit.client.0.trial112.stderr:218471: start osd.9 2026-02-13T02:08:40.064 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:08:41.546 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2026-02-13T02:08:41.546 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2026-02-13T02:08:41.546 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:08:41.546 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:08:41.547 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:08:41.547 INFO:tasks.workunit.client.0.trial112.stderr:218471: 2026-02-13T02:08:40.074+0000 7faab7405a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:08:41.547 INFO:tasks.workunit.client.0.trial112.stderr:218471: 2026-02-13T02:08:40.074+0000 7faab7405a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:08:41.547 INFO:tasks.workunit.client.0.trial112.stderr:218471: 2026-02-13T02:08:40.078+0000 7faab7405a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:08:41.547 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2026-02-13T02:08:41.547 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:08:41.547 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2026-02-13T02:08:41.547 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:08:41.547 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:08:41.547 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:08:41.547 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:08:41.547 INFO:tasks.workunit.client.0.trial112.stderr:218471: 0 2026-02-13T02:08:41.548 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:08:41.548 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-13T02:08:41.548 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:08:41.548 INFO:tasks.workunit.client.0.trial112.stderr:218471: 2026-02-13T02:08:40.542+0000 7faab7405a40 -1 Falling back to public interface 2026-02-13T02:08:41.548 INFO:tasks.workunit.client.0.trial112.stderr:218471: 2026-02-13T02:08:41.214+0000 7faab7405a40 -1 osd.9 84 log_to_monitors true 2026-02-13T02:08:41.548 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:08:41.548 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:08:41.548 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:08:41.548 INFO:tasks.workunit.client.0.trial112.stderr:218471: 1 2026-02-13T02:08:41.548 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:08:41.548 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-13T02:08:41.548 INFO:tasks.workunit.client.0.trial112.stderr:218471: osd.9 up in weight 1 up_from 89 up_thru 0 down_at 85 last_clean_interval [83,84) [v2:127.0.0.1:6874/1743157986,v1:127.0.0.1:6875/1743157986] [v2:127.0.0.1:6876/1743157986,v1:127.0.0.1:6877/1743157986] exists,up 18292385-2986-4859-a36e-1a5e00d7be2a 2026-02-13T02:08:41.549 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:08:41.549 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:08:41.549 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:08:41.549 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:08:41.549 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:08:41.549 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:08:41.549 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:08:41.549 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:08:41.549 INFO:tasks.workunit.client.0.trial112.stderr:218471: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:08:41.553 INFO:tasks.workunit.client.0.trial112.stderr:218471: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: ge18469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2026-02-13T02:08:41.553 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2026-02-13T02:08:41.553 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:08:41.553 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:08:41.553 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:08:41.553 INFO:tasks.workunit.client.0.trial112.stderr:218469: 2026-02-13T02:08:40.062+0000 7f28447eba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:08:41.553 INFO:tasks.workunit.client.0.trial112.stderr:218469: 2026-02-13T02:08:40.066+0000 7f28447eba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:08:41.553 INFO:tasks.workunit.client.0.trial112.stderr:218469: 2026-02-13T02:08:40.066+0000 7f28447eba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:08:41.554 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2026-02-13T02:08:41.554 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:08:41.554 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-13T02:08:41.554 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:08:41.554 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:08:41.554 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:08:41.554 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:08:41.554 INFO:tasks.workunit.client.0.trial112.stderr:218469: 0 2026-02-13T02:08:41.554 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:08:41.554 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-13T02:08:41.554 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:08:41.554 INFO:tasks.workunit.client.0.trial112.stderr:218469: 2026-02-13T02:08:40.522+0000 7f28447eba40 -1 Falling back to public interface 2026-02-13T02:08:41.555 INFO:tasks.workunit.client.0.trial112.stderr:218469: 2026-02-13T02:08:41.194+0000 7f28447eba40 -1 osd.5 84 log_to_monitors true 2026-02-13T02:08:41.555 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:08:41.555 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:08:41.555 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:08:41.555 INFO:tasks.workunit.client.0.trial112.stderr:218469: 1 2026-02-13T02:08:41.555 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:08:41.555 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-13T02:08:41.555 INFO:tasks.workunit.client.0.trial112.stderr:218469: osd.5 up in weight 1 up_from 89 up_thru 89 down_at 85 last_clean_interval [77,84) [v2:127.0.0.1:6842/3211259131,v1:127.0.0.1:6843/3211259131] [v2:127.0.0.1:6844/3211259131,v1:127.0.0.1:6845/3211259131] exists,up 03eef3ce-a0e3-4fec-86e3-c40fb660505f 2026-02-13T02:08:41.555 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:08:41.555 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:08:41.555 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:08:41.555 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:08:41.556 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:08:41.556 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:08:41.556 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:08:41.556 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:08:41.556 INFO:tasks.workunit.client.0.trial112.stderr:218469: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:08:42.116 INFO:tasks.workunit.client.0.trial112.stderr:218469: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: gt_timeout_delays: echo true 2026-02-13T02:08:42.116 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:08:42.116 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:08:42.116 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:08:42.116 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:08:42.117 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:08:42.117 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:08:42.117 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:08:42.117 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:08:42.117 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:08:42.117 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:08:42.117 INFO:tasks.workunit.client.0.trial112.stderr:218471: 1 2026-02-13T02:08:42.117 INFO:tasks.workunit.client.0.trial112.stderr:218471: 2 2026-02-13T02:08:42.117 INFO:tasks.workunit.client.0.trial112.stderr:218471: 3 2026-02-13T02:08:42.117 INFO:tasks.workunit.client.0.trial112.stderr:218471: 4 2026-02-13T02:08:42.117 INFO:tasks.workunit.client.0.trial112.stderr:218471: 5 2026-02-13T02:08:42.117 INFO:tasks.workunit.client.0.trial112.stderr:218471: 6 2026-02-13T02:08:42.118 INFO:tasks.workunit.client.0.trial112.stderr:218471: 7 2026-02-13T02:08:42.118 INFO:tasks.workunit.client.0.trial112.stderr:218471: 8 2026-02-13T02:08:42.118 INFO:tasks.workunit.client.0.trial112.stderr:218471: 9' 2026-02-13T02:08:42.118 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:08:42.118 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:42.118 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:08:42.118 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836504 2026-02-13T02:08:42.118 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836504 2026-02-13T02:08:42.118 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504' 2026-02-13T02:08:42.118 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:42.118 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:08:42.118 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705688 2026-02-13T02:08:42.118 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705688 2026-02-13T02:08:42.119 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-38654705688' 2026-02-13T02:08:42.119 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:42.119 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:08:42.119 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542167 2026-02-13T02:08:42.119 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542167 2026-02-13T02:08:42.119 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-38654705688 2-60129542167' 2026-02-13T02:08:42.119 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:42.119 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:08:42.119 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=279172874253 2026-02-13T02:08:42.119 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 279172874253 2026-02-13T02:08:42.119 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-38654705688 2-60129542167 3-279172874253' 2026-02-13T02:08:42.119 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:42.120 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: et_timeout_delays: echo true 2026-02-13T02:08:42.120 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:08:42.120 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:08:42.120 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:08:42.120 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:08:42.120 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:08:42.121 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:08:42.121 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:08:42.121 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:08:42.121 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:08:42.121 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:08:42.121 INFO:tasks.workunit.client.0.trial112.stderr:218469: 1 2026-02-13T02:08:42.121 INFO:tasks.workunit.client.0.trial112.stderr:218469: 2 2026-02-13T02:08:42.121 INFO:tasks.workunit.client.0.trial112.stderr:218469: 3 2026-02-13T02:08:42.121 INFO:tasks.workunit.client.0.trial112.stderr:218469: 4 2026-02-13T02:08:42.121 INFO:tasks.workunit.client.0.trial112.stderr:218469: 5 2026-02-13T02:08:42.121 INFO:tasks.workunit.client.0.trial112.stderr:218469: 6 2026-02-13T02:08:42.121 INFO:tasks.workunit.client.0.trial112.stderr:218469: 7 2026-02-13T02:08:42.122 INFO:tasks.workunit.client.0.trial112.stderr:218469: 8 2026-02-13T02:08:42.122 INFO:tasks.workunit.client.0.trial112.stderr:218469: 9' 2026-02-13T02:08:42.122 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:08:42.122 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:42.122 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:08:42.122 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836505 2026-02-13T02:08:42.122 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836505 2026-02-13T02:08:42.122 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505' 2026-02-13T02:08:42.122 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:42.122 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:08:42.122 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705689 2026-02-13T02:08:42.122 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705689 2026-02-13T02:08:42.122 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-38654705689' 2026-02-13T02:08:42.123 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:42.123 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:08:42.123 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542168 2026-02-13T02:08:42.123 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542168 2026-02-13T02:08:42.123 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-38654705689 2-60129542168' 2026-02-13T02:08:42.123 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:42.123 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:08:42.123 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=279172874254 2026-02-13T02:08:42.123 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 279172874254 2026-02-13T02:08:42.123 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-38654705689 2-60129542168 3-279172874254' 2026-02-13T02:08:42.123 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:42.557 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258:flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-13T02:08:42.557 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280533 2026-02-13T02:08:42.558 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280533 2026-02-13T02:08:42.558 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-38654705688 2-60129542167 3-279172874253 4-94489280533' 2026-02-13T02:08:42.558 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:42.558 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-13T02:08:42.558 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=382252089346 2026-02-13T02:08:42.558 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 382252089346 2026-02-13T02:08:42.558 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-38654705688 2-60129542167 3-279172874253 4-94489280533 5-382252089346' 2026-02-13T02:08:42.558 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:42.558 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-13T02:08:42.558 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986196 2026-02-13T02:08:42.558 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986196 2026-02-13T02:08:42.559 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-38654705688 2-60129542167 3-279172874253 4-94489280533 5-382252089346 6-133143986196' 2026-02-13T02:08:42.559 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:42.559 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-13T02:08:42.559 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855379 2026-02-13T02:08:42.559 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855379 2026-02-13T02:08:42.559 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-38654705688 2-60129542167 3-279172874253 4-94489280533 5-382252089346 6-133143986196 7-150323855379' 2026-02-13T02:08:42.559 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:42.559 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-13T02:08:42.559 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691859 2026-02-13T02:08:42.559 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691859 2026-02-13T02:08:42.559 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-38654705688 2-60129542167 3-279172874253 4-94489280533 5-382252089346 6-133143986196 7-150323855379 8-171798691859' 2026-02-13T02:08:42.559 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:42.560 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-13T02:08:42.560 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=382252089346 2026-02-13T02:08:42.560 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 382252089346 2026-02-13T02:08:42.560 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-38654705688 2-60129542167 3-279172874253 4-94489280533 5-382252089346 6-133143986196 7-150323855379 8-171798691859 9-382252089346' 2026-02-13T02:08:42.560 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:42.560 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/ceph flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-13T02:08:42.561 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280534 2026-02-13T02:08:42.561 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280534 2026-02-13T02:08:42.561 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-38654705689 2-60129542168 3-279172874254 4-94489280534' 2026-02-13T02:08:42.561 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:42.561 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-13T02:08:42.561 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=382252089347 2026-02-13T02:08:42.561 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 382252089347 2026-02-13T02:08:42.561 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-38654705689 2-60129542168 3-279172874254 4-94489280534 5-382252089347' 2026-02-13T02:08:42.561 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:42.561 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-13T02:08:42.561 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986197 2026-02-13T02:08:42.561 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986197 2026-02-13T02:08:42.562 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-38654705689 2-60129542168 3-279172874254 4-94489280534 5-382252089347 6-133143986197' 2026-02-13T02:08:42.562 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:42.562 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-13T02:08:42.562 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855380 2026-02-13T02:08:42.562 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855380 2026-02-13T02:08:42.562 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-38654705689 2-60129542168 3-279172874254 4-94489280534 5-382252089347 6-133143986197 7-150323855380' 2026-02-13T02:08:42.562 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:42.562 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-13T02:08:42.562 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691860 2026-02-13T02:08:42.562 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691860 2026-02-13T02:08:42.562 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-38654705689 2-60129542168 3-279172874254 4-94489280534 5-382252089347 6-133143986197 7-150323855380 8-171798691860' 2026-02-13T02:08:42.562 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:42.562 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-13T02:08:42.563 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=382252089347 2026-02-13T02:08:42.563 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 382252089347 2026-02-13T02:08:42.563 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-38654705689 2-60129542168 3-279172874254 4-94489280534 5-382252089347 6-133143986197 7-150323855380 8-171798691860 9-382252089347' 2026-02-13T02:08:42.563 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:44.233 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/ceptest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836504 2026-02-13T02:08:44.234 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:44.234 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:08:44.234 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836504 2026-02-13T02:08:44.234 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:44.234 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836504 2026-02-13T02:08:44.234 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2026-02-13T02:08:44.234 INFO:tasks.workunit.client.0.trial112.stderr:218471: waiting osd.0 seq 21474836504 2026-02-13T02:08:44.234 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:08:44.234 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836503 -lt 21474836504 2026-02-13T02:08:44.235 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:08:44.235 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:08:44.235 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:08:44.235 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836505 -lt 21474836504 2026-02-13T02:08:44.235 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:44.235 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705688 2026-02-13T02:08:44.235 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:44.235 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:08:44.235 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705688 2026-02-13T02:08:44.235 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:44.235 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705688 2026-02-13T02:08:44.236 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705688' 2026-02-13T02:08:44.236 INFO:tasks.workunit.client.0.trial112.stderr:218471: waiting osd.1 seq 38654705688 2026-02-13T02:08:44.236 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:08:44.236 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705689 -lt 38654705688 2026-02-13T02:08:44.236 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:44.236 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542167 2026-02-13T02:08:44.236 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:44.236 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:08:44.236 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542167 2026-02-13T02:08:44.236 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:44.236 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542167 2026-02-13T02:08:44.236 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542167' 2026-02-13T02:08:44.236 INFO:tasks.workunit.client.0.trial112.stderr:218471: waiting osd.2 seq 60129542167 2026-02-13T02:08:44.237 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:08:44.237 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542168 -lt 60129542167 2026-02-13T02:08:44.241 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_statshtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836505 2026-02-13T02:08:44.241 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:44.241 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:08:44.241 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836505 2026-02-13T02:08:44.241 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:44.241 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836505 2026-02-13T02:08:44.241 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2026-02-13T02:08:44.241 INFO:tasks.workunit.client.0.trial112.stderr:218469: waiting osd.0 seq 21474836505 2026-02-13T02:08:44.241 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:08:44.241 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836503 -lt 21474836505 2026-02-13T02:08:44.241 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:08:44.242 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:08:44.242 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:08:44.242 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836505 -lt 21474836505 2026-02-13T02:08:44.242 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:44.242 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705689 2026-02-13T02:08:44.242 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:44.242 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:08:44.242 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705689 2026-02-13T02:08:44.242 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:44.242 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705689 2026-02-13T02:08:44.242 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705689' 2026-02-13T02:08:44.242 INFO:tasks.workunit.client.0.trial112.stderr:218469: waiting osd.1 seq 38654705689 2026-02-13T02:08:44.242 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:08:44.243 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705689 -lt 38654705689 2026-02-13T02:08:44.243 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:44.243 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542168 2026-02-13T02:08:44.243 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:44.243 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:08:44.243 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542168 2026-02-13T02:08:44.243 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:44.243 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542168 2026-02-13T02:08:44.243 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542168' 2026-02-13T02:08:44.243 INFO:tasks.workunit.client.0.trial112.stderr:218469: waiting osd.2 seq 60129542168 2026-02-13T02:08:44.243 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:08:44.243 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542168 -lt 60129542168 2026-02-13T02:08:44.737 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stat: for s in $seqs 2026-02-13T02:08:44.738 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-279172874253 2026-02-13T02:08:44.738 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:44.738 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:08:44.738 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-279172874253 2026-02-13T02:08:44.738 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:44.738 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=279172874253 2026-02-13T02:08:44.738 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 279172874253' 2026-02-13T02:08:44.738 INFO:tasks.workunit.client.0.trial112.stderr:218471: waiting osd.3 seq 279172874253 2026-02-13T02:08:44.738 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:08:44.739 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 279172874254 -lt 279172874253 2026-02-13T02:08:44.739 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:44.739 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-94489280533 2026-02-13T02:08:44.739 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:44.739 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-13T02:08:44.739 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-94489280533 2026-02-13T02:08:44.739 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:44.739 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280533 2026-02-13T02:08:44.739 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 94489280533' 2026-02-13T02:08:44.739 INFO:tasks.workunit.client.0.trial112.stderr:218471: waiting osd.4 seq 94489280533 2026-02-13T02:08:44.739 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-13T02:08:44.739 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280535 -lt 94489280533 2026-02-13T02:08:44.740 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:44.740 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-382252089346 2026-02-13T02:08:44.740 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:44.740 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-13T02:08:44.740 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-382252089346 2026-02-13T02:08:44.740 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:44.740 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=382252089346 2026-02-13T02:08:44.740 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 382252089346' 2026-02-13T02:08:44.740 INFO:tasks.workunit.client.0.trial112.stderr:218471: waiting osd.5 seq 382252089346 2026-02-13T02:08:44.740 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-13T02:08:44.740 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 382252089347 -lt 382252089346 2026-02-13T02:08:44.740 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:44.740 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-133143986196 2026-02-13T02:08:44.741 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:44.741 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-13T02:08:44.760 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-s: for s in $seqs 2026-02-13T02:08:44.760 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-279172874254 2026-02-13T02:08:44.760 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:44.760 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:08:44.760 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-279172874254 2026-02-13T02:08:44.760 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:44.760 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=279172874254 2026-02-13T02:08:44.761 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 279172874254' 2026-02-13T02:08:44.761 INFO:tasks.workunit.client.0.trial112.stderr:218469: waiting osd.3 seq 279172874254 2026-02-13T02:08:44.761 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:08:44.761 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 279172874254 -lt 279172874254 2026-02-13T02:08:44.761 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:44.761 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-94489280534 2026-02-13T02:08:44.761 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:44.761 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-13T02:08:44.761 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-94489280534 2026-02-13T02:08:44.761 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:44.761 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280534 2026-02-13T02:08:44.761 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 94489280534' 2026-02-13T02:08:44.762 INFO:tasks.workunit.client.0.trial112.stderr:218469: waiting osd.4 seq 94489280534 2026-02-13T02:08:44.762 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-13T02:08:44.762 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280535 -lt 94489280534 2026-02-13T02:08:44.762 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:44.762 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-382252089347 2026-02-13T02:08:44.762 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:44.762 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-13T02:08:44.762 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-382252089347 2026-02-13T02:08:44.762 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:44.762 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=382252089347 2026-02-13T02:08:44.762 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 382252089347' 2026-02-13T02:08:44.762 INFO:tasks.workunit.client.0.trial112.stderr:218469: waiting osd.5 seq 382252089347 2026-02-13T02:08:44.763 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-13T02:08:44.763 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 382252089347 -lt 382252089347 2026-02-13T02:08:44.763 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:44.763 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-133143986197 2026-02-13T02:08:44.763 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:44.763 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-13T02:08:45.231 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/cephhelpers.sh:2268: flush_pg_stats: echo 6-133143986196 2026-02-13T02:08:45.231 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:45.232 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=133143986196 2026-02-13T02:08:45.232 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 133143986196' 2026-02-13T02:08:45.232 INFO:tasks.workunit.client.0.trial112.stderr:218471: waiting osd.6 seq 133143986196 2026-02-13T02:08:45.232 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-13T02:08:45.232 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 133143986197 -lt 133143986196 2026-02-13T02:08:45.232 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:45.232 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-150323855379 2026-02-13T02:08:45.232 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:45.232 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-13T02:08:45.232 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-150323855379 2026-02-13T02:08:45.232 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:45.233 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855379 2026-02-13T02:08:45.233 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 150323855379' 2026-02-13T02:08:45.233 INFO:tasks.workunit.client.0.trial112.stderr:218471: waiting osd.7 seq 150323855379 2026-02-13T02:08:45.233 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-13T02:08:45.233 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855381 -lt 150323855379 2026-02-13T02:08:45.233 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:45.233 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-171798691859 2026-02-13T02:08:45.233 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:45.233 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-13T02:08:45.233 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-171798691859 2026-02-13T02:08:45.233 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:45.234 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691859 2026-02-13T02:08:45.234 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 171798691859' 2026-02-13T02:08:45.234 INFO:tasks.workunit.client.0.trial112.stderr:218471: waiting osd.8 seq 171798691859 2026-02-13T02:08:45.234 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-13T02:08:45.234 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691860 -lt 171798691859 2026-02-13T02:08:45.234 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:45.234 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-382252089346 2026-02-13T02:08:45.234 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:45.234 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-13T02:08:45.234 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-382252089346 2026-02-13T02:08:45.234 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:45.234 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=382252089346 2026-02-13T02:08:45.273 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 38225208-helpers.sh:2268: flush_pg_stats: echo 6-133143986197 2026-02-13T02:08:45.273 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:45.273 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=133143986197 2026-02-13T02:08:45.273 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 133143986197' 2026-02-13T02:08:45.273 INFO:tasks.workunit.client.0.trial112.stderr:218469: waiting osd.6 seq 133143986197 2026-02-13T02:08:45.273 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-13T02:08:45.273 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 133143986197 -lt 133143986197 2026-02-13T02:08:45.274 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:45.274 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-150323855380 2026-02-13T02:08:45.274 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:45.274 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-13T02:08:45.274 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-150323855380 2026-02-13T02:08:45.274 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:45.274 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855380 2026-02-13T02:08:45.274 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 150323855380' 2026-02-13T02:08:45.274 INFO:tasks.workunit.client.0.trial112.stderr:218469: waiting osd.7 seq 150323855380 2026-02-13T02:08:45.274 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-13T02:08:45.274 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855381 -lt 150323855380 2026-02-13T02:08:45.275 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:45.275 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-171798691860 2026-02-13T02:08:45.275 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:45.275 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-13T02:08:45.275 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-171798691860 2026-02-13T02:08:45.275 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:45.275 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691860 2026-02-13T02:08:45.275 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 171798691860' 2026-02-13T02:08:45.275 INFO:tasks.workunit.client.0.trial112.stderr:218469: waiting osd.8 seq 171798691860 2026-02-13T02:08:45.275 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-13T02:08:45.276 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691860 -lt 171798691860 2026-02-13T02:08:45.276 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:45.276 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-382252089347 2026-02-13T02:08:45.276 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:45.276 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-13T02:08:45.276 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-382252089347 2026-02-13T02:08:45.276 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:45.276 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=382252089347 2026-02-13T02:08:45.970 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 38225209346' 2026-02-13T02:08:45.971 INFO:tasks.workunit.client.0.trial112.stderr:218471: waiting osd.9 seq 382252089346 2026-02-13T02:08:45.971 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-13T02:08:45.971 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 382252089347 -lt 382252089346 2026-02-13T02:08:45.971 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:08:45.971 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:08:45.971 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:08:45.971 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:08:45.971 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:08:45.971 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:08:45.971 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:08:45.972 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:08:45.972 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:08:45.972 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:08:45.972 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:08:45.972 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:08:45.972 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:08:45.972 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:08:45.972 INFO:tasks.workunit.client.0.trial112.stderr:218471: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:08:45.972 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:08:45.972 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:08:45.972 INFO:tasks.workunit.client.0.trial112.stderr:218471: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:08:45.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-13T02:08:45.991 INFO:tasks.workunit.client.0.trial112.stderr:89347' 2026-02-13T02:08:45.992 INFO:tasks.workunit.client.0.trial112.stderr:218469: waiting osd.9 seq 382252089347 2026-02-13T02:08:45.992 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-13T02:08:45.992 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 382252089347 -lt 382252089347 2026-02-13T02:08:45.992 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:08:45.992 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:08:45.992 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:08:45.992 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:08:45.992 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:08:45.992 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:08:45.992 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:08:45.992 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:08:45.993 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:08:45.993 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:08:45.993 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:08:45.993 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:08:45.993 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:08:45.993 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:08:45.993 INFO:tasks.workunit.client.0.trial112.stderr:218469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:08:45.993 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:08:45.993 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:08:45.993 INFO:tasks.workunit.client.0.trial112.stderr:218469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:08:45.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-13T02:08:45.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-13T02:08:45.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 218468 2026-02-13T02:08:45.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-13T02:08:45.995 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-13T02:08:45.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-13T02:08:45.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:257: corrupt_and_repair_two: return_code=0 2026-02-13T02:08:45.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:258: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-13T02:08:45.996 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: get_pg ecpool SOMETHING 2026-02-13T02:08:45.996 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-13T02:08:45.996 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-13T02:08:45.996 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-13T02:08:45.996 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T02:08:46.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: local pg=2.0 2026-02-13T02:08:46.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:264: corrupt_and_repair_two: repair 2.0 2026-02-13T02:08:46.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-13T02:08:46.153 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-13T02:08:46.153 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:08:46.153 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:08:46.154 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:08:46.154 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:08:46.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-13T02:08:30.289141+0000 2026-02-13T02:08:46.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-13T02:08:46.466 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-13T02:08:46.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-13T02:08:30.289141+0000 2026-02-13T02:08:46.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-13T02:08:46.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:08:30.289141+0000 2026-02-13T02:08:46.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:08:46.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:08:46.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:08:46.478 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:08:46.478 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:08:46.478 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:08:46.478 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:08:46.478 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:08:46.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:08:30.289141+0000 '>' 2026-02-13T02:08:30.289141+0000 2026-02-13T02:08:46.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:08:47.642 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:08:47.642 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:08:47.643 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:08:47.643 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:08:47.643 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:08:47.644 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:08:47.644 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:08:47.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:08:30.289141+0000 '>' 2026-02-13T02:08:30.289141+0000 2026-02-13T02:08:47.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:08:48.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:08:48.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:08:48.804 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:08:48.805 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:08:48.805 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:08:48.805 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:08:48.805 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:08:48.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:08:30.289141+0000 '>' 2026-02-13T02:08:30.289141+0000 2026-02-13T02:08:48.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:08:49.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:08:49.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:08:49.966 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:08:49.966 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:08:49.966 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:08:49.966 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:08:49.966 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:08:50.128 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:08:30.289141+0000 '>' 2026-02-13T02:08:30.289141+0000 2026-02-13T02:08:50.128 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:08:51.129 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:08:51.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:08:51.131 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:08:51.131 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:08:51.131 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:08:51.131 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:08:51.131 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:08:51.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:08:30.289141+0000 '>' 2026-02-13T02:08:30.289141+0000 2026-02-13T02:08:51.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:08:52.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:08:52.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:08:52.299 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:08:52.299 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:08:52.299 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:08:52.299 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:08:52.299 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:08:52.465 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:08:30.289141+0000 '>' 2026-02-13T02:08:30.289141+0000 2026-02-13T02:08:52.465 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:08:53.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:08:53.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:08:53.467 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:08:53.467 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:08:53.467 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:08:53.467 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:08:53.468 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:08:53.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:08:47.347459+0000 '>' 2026-02-13T02:08:30.289141+0000 2026-02-13T02:08:53.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:08:53.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: corrupt_and_repair_two: pids= 2026-02-13T02:08:53.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-13T02:08:53.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-13T02:08:53.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-13T02:08:53.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 221928"' 2026-02-13T02:08:53.634 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 221928' 2026-02-13T02:08:53.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-13T02:08:53.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-13T02:08:53.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-13T02:08:53.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 221929"' 2026-02-13T02:08:53.634 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 221929' 2026-02-13T02:08:53.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:271: corrupt_and_repair_two: wait_background pids 2026-02-13T02:08:53.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 221928 221929' 2026-02-13T02:08:53.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-13T02:08:53.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-13T02:08:53.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 221928 2026-02-13T02:08:53.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-13T02:08:53.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/221931: /' 2026-02-13T02:08:53.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-13T02:08:53.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/221933: /' 2026-02-13T02:08:54.174 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:08:54.174 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:08:54.174 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2026-02-13T02:08:54.175 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:08:54.175 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-13T02:08:54.175 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:08:54.175 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:08:54.175 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2026-02-13T02:08:54.175 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:08:54.175 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2026-02-13T02:08:54.175 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:08:54.175 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:08:54.175 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:08:54.175 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:08:54.175 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:08:54.176 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:08:54.176 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-13T02:08:54.176 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:08:54.176 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:08:54.176 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2026-02-13T02:08:54.176 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:08:54.176 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2026-02-13T02:08:54.176 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2026-02-13T02:08:54.176 INFO:tasks.workunit.client.0.trial112.stderr:221931: _ 2026-02-13T02:08:54.176 INFO:tasks.workunit.client.0.trial112.stderr:221931: hinfo_key 2026-02-13T02:08:54.176 INFO:tasks.workunit.client.0.trial112.stderr:221931: snapset 2026-02-13T02:08:54.176 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2026-02-13T02:08:54.177 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:08:54.177 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:08:54.177 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2026-02-13T02:08:54.177 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:08:54.177 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2026-02-13T02:08:54.177 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:08:54.177 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:08:54.177 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:08:54.177 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:08:54.177 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2026-02-13T02:08:54.177 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:08:54.177 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-13T02:08:54.178 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:08:54.178 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:08:54.178 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2026-02-13T02:08:54.178 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:08:54.178 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2026-02-13T02:08:54.178 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:08:54.178 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:08:54.178 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:08:54.178 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:08:54.178 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:08:54.178 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:08:54.178 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-13T02:08:54.178 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:08:54.179 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:08:54.179 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2026-02-13T02:08:54.179 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:08:54.179 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2026-02-13T02:08:54.179 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2026-02-13T02:08:54.179 INFO:tasks.workunit.client.0.trial112.stderr:221933: _ 2026-02-13T02:08:54.179 INFO:tasks.workunit.client.0.trial112.stderr:221933: hinfo_key 2026-02-13T02:08:54.179 INFO:tasks.workunit.client.0.trial112.stderr:221933: snapset 2026-02-13T02:08:54.179 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2026-02-13T02:08:54.179 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:08:54.179 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:08:54.179 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2026-02-13T02:08:54.180 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:08:54.180 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2026-02-13T02:08:54.180 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:08:54.180 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:08:54.181 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:08:54.181 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:08:54.181 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2026-02-13T02:08:54.181 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2026-02-13T02:08:54.181 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:08:54.182 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:08:54.182 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:08:54.182 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:08:54.182 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:08:54.182 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:08:54.182 INFO:tasks.workunit.client.0.trial112.stderr:221931: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:08:54.182 INFO:tasks.workunit.client.0.trial112.stderr:221931: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:08:54.182 INFO:tasks.workunit.client.0.trial112.stderr:221931: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:08:54.182 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:08:54.182 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:08:54.182 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:08:54.182 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:08:54.183 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:08:54.183 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:08:54.183 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:08:54.183 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:08:54.183 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:08:54.183 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:08:54.183 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:08:54.183 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2026-02-13T02:08:54.183 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2026-02-13T02:08:54.183 INFO:tasks.workunit.client.0.trial112.stderr:221931: start osd.5 2026-02-13T02:08:54.183 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recos.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:08:54.183 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:08:54.183 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2026-02-13T02:08:54.184 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2026-02-13T02:08:54.184 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:08:54.184 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:08:54.184 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:08:54.184 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:08:54.184 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:08:54.184 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:08:54.184 INFO:tasks.workunit.client.0.trial112.stderr:221933: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:08:54.184 INFO:tasks.workunit.client.0.trial112.stderr:221933: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:08:54.184 INFO:tasks.workunit.client.0.trial112.stderr:221933: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:08:54.184 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:08:54.184 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:08:54.184 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:08:54.185 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:08:54.185 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:08:54.185 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:08:54.185 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:08:54.185 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:08:54.185 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:08:54.185 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:08:54.185 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:08:54.185 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2026-02-13T02:08:54.185 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2026-02-13T02:08:54.185 INFO:tasks.workunit.client.0.trial112.stderr:221933: start osd.9 2026-02-13T02:08:55.667 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:08:55.667 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2026-02-13T02:08:55.668 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2026-02-13T02:08:55.668 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:08:55.668 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:08:55.668 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:08:55.668 INFO:tasks.workunit.client.0.trial112.stderr:221933: 2026-02-13T02:08:54.182+0000 7fc7187f3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:08:55.668 INFO:tasks.workunit.client.0.trial112.stderr:221933: 2026-02-13T02:08:54.186+0000 7fc7187f3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:08:55.668 INFO:tasks.workunit.client.0.trial112.stderr:221933: 2026-02-13T02:08:54.186+0000 7fc7187f3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:08:55.668 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2026-02-13T02:08:55.668 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:08:55.668 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2026-02-13T02:08:55.669 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:08:55.669 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:08:55.669 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:08:55.669 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:08:55.669 INFO:tasks.workunit.client.0.trial112.stderr:221933: 0 2026-02-13T02:08:55.669 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:08:55.669 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-13T02:08:55.669 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:08:55.669 INFO:tasks.workunit.client.0.trial112.stderr:221933: 2026-02-13T02:08:54.610+0000 7fc7187f3a40 -1 Falling back to public interface 2026-02-13T02:08:55.669 INFO:tasks.workunit.client.0.trial112.stderr:221933: 2026-02-13T02:08:55.282+0000 7fc7187f3a40 -1 osd.9 90 log_to_monitors true 2026-02-13T02:08:55.669 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:08:55.669 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:08:55.670 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:08:55.670 INFO:tasks.workunit.client.0.trial112.stderr:221933: 1 2026-02-13T02:08:55.670 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:08:55.670 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-13T02:08:55.670 INFO:tasks.workunit.client.0.trial112.stderr:221933: osd.9 up in weight 1 up_from 94 up_thru 0 down_at 91 last_clean_interval [89,90) [v2:127.0.0.1:6842/750193657,v1:127.0.0.1:6843/750193657] [v2:127.0.0.1:6844/750193657,v1:127.0.0.1:6845/750193657] exists,up 18292385-2986-4859-a36e-1a5e00d7be2a 2026-02-13T02:08:55.670 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:08:55.670 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:08:55.670 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:08:55.670 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:08:55.670 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:08:55.670 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:08:55.670 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:08:55.670 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:08:55.671 INFO:tasks.workunit.client.0.trial112.stderr:221933: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:08:55.679 INFO:tasks.workunit.client.0.trial112.stderr:221933: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:15very_ops 2026-02-13T02:08:55.679 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2026-02-13T02:08:55.679 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2026-02-13T02:08:55.679 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:08:55.679 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:08:55.679 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:08:55.680 INFO:tasks.workunit.client.0.trial112.stderr:221931: 2026-02-13T02:08:54.182+0000 7efd39074a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:08:55.680 INFO:tasks.workunit.client.0.trial112.stderr:221931: 2026-02-13T02:08:54.186+0000 7efd39074a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:08:55.680 INFO:tasks.workunit.client.0.trial112.stderr:221931: 2026-02-13T02:08:54.186+0000 7efd39074a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:08:55.680 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2026-02-13T02:08:55.680 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:08:55.680 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-13T02:08:55.680 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:08:55.680 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:08:55.680 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:08:55.680 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:08:55.680 INFO:tasks.workunit.client.0.trial112.stderr:221931: 0 2026-02-13T02:08:55.680 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:08:55.680 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-13T02:08:55.681 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:08:55.681 INFO:tasks.workunit.client.0.trial112.stderr:221931: 2026-02-13T02:08:54.634+0000 7efd39074a40 -1 Falling back to public interface 2026-02-13T02:08:55.681 INFO:tasks.workunit.client.0.trial112.stderr:221931: 2026-02-13T02:08:55.334+0000 7efd39074a40 -1 osd.5 90 log_to_monitors true 2026-02-13T02:08:55.681 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:08:55.681 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:08:55.681 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:08:55.681 INFO:tasks.workunit.client.0.trial112.stderr:221931: 1 2026-02-13T02:08:55.681 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:08:55.681 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-13T02:08:55.681 INFO:tasks.workunit.client.0.trial112.stderr:221931: osd.5 up in weight 1 up_from 94 up_thru 94 down_at 91 last_clean_interval [89,90) [v2:127.0.0.1:6874/157293920,v1:127.0.0.1:6875/157293920] [v2:127.0.0.1:6876/157293920,v1:127.0.0.1:6877/157293920] exists,up 03eef3ce-a0e3-4fec-86e3-c40fb660505f 2026-02-13T02:08:55.681 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:08:55.681 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:08:55.681 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:08:55.682 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:08:55.682 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:08:55.682 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:08:55.682 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:08:55.682 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:08:55.682 INFO:tasks.workunit.client.0.trial112.stderr:221931: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:08:56.218 INFO:tasks.workunit.client.0.trial112.stderr:221931: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:193: get_timeout_delays: echo true 2026-02-13T02:08:56.218 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:08:56.218 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:08:56.218 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:08:56.218 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:08:56.218 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:08:56.218 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:08:56.218 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:08:56.218 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:08:56.219 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:08:56.219 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:08:56.219 INFO:tasks.workunit.client.0.trial112.stderr:221933: 1 2026-02-13T02:08:56.219 INFO:tasks.workunit.client.0.trial112.stderr:221933: 2 2026-02-13T02:08:56.219 INFO:tasks.workunit.client.0.trial112.stderr:221933: 3 2026-02-13T02:08:56.219 INFO:tasks.workunit.client.0.trial112.stderr:221933: 4 2026-02-13T02:08:56.219 INFO:tasks.workunit.client.0.trial112.stderr:221933: 5 2026-02-13T02:08:56.219 INFO:tasks.workunit.client.0.trial112.stderr:221933: 6 2026-02-13T02:08:56.219 INFO:tasks.workunit.client.0.trial112.stderr:221933: 7 2026-02-13T02:08:56.219 INFO:tasks.workunit.client.0.trial112.stderr:221933: 8 2026-02-13T02:08:56.219 INFO:tasks.workunit.client.0.trial112.stderr:221933: 9' 2026-02-13T02:08:56.219 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:08:56.220 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:56.220 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:08:56.220 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836509 2026-02-13T02:08:56.220 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836509 2026-02-13T02:08:56.220 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509' 2026-02-13T02:08:56.220 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:56.220 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:08:56.220 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705692 2026-02-13T02:08:56.220 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705692 2026-02-13T02:08:56.220 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-38654705692' 2026-02-13T02:08:56.220 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:56.220 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:08:56.220 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542172 2026-02-13T02:08:56.221 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542172 2026-02-13T02:08:56.221 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-38654705692 2-60129542172' 2026-02-13T02:08:56.221 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:56.221 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:08:56.221 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=279172874257 2026-02-13T02:08:56.221 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 279172874257 2026-02-13T02:08:56.221 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-38654705692 2-60129542172 3-279172874257' 2026-02-13T02:08:56.221 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:56.248 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:593: get_timeout_delays: echo true 2026-02-13T02:08:56.248 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:08:56.248 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:08:56.248 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:08:56.248 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:08:56.248 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:08:56.248 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:08:56.248 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:08:56.248 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:08:56.248 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:08:56.249 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:08:56.249 INFO:tasks.workunit.client.0.trial112.stderr:221931: 1 2026-02-13T02:08:56.249 INFO:tasks.workunit.client.0.trial112.stderr:221931: 2 2026-02-13T02:08:56.249 INFO:tasks.workunit.client.0.trial112.stderr:221931: 3 2026-02-13T02:08:56.249 INFO:tasks.workunit.client.0.trial112.stderr:221931: 4 2026-02-13T02:08:56.249 INFO:tasks.workunit.client.0.trial112.stderr:221931: 5 2026-02-13T02:08:56.249 INFO:tasks.workunit.client.0.trial112.stderr:221931: 6 2026-02-13T02:08:56.249 INFO:tasks.workunit.client.0.trial112.stderr:221931: 7 2026-02-13T02:08:56.249 INFO:tasks.workunit.client.0.trial112.stderr:221931: 8 2026-02-13T02:08:56.249 INFO:tasks.workunit.client.0.trial112.stderr:221931: 9' 2026-02-13T02:08:56.249 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:08:56.249 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:56.249 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:08:56.250 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836510 2026-02-13T02:08:56.250 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836510 2026-02-13T02:08:56.250 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510' 2026-02-13T02:08:56.250 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:56.250 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:08:56.250 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705693 2026-02-13T02:08:56.250 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705693 2026-02-13T02:08:56.250 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-38654705693' 2026-02-13T02:08:56.250 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:56.250 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:08:56.250 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542173 2026-02-13T02:08:56.250 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542173 2026-02-13T02:08:56.251 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-38654705693 2-60129542173' 2026-02-13T02:08:56.251 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:56.251 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:08:56.251 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=279172874258 2026-02-13T02:08:56.251 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 279172874258 2026-02-13T02:08:56.251 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-38654705693 2-60129542173 3-279172874258' 2026-02-13T02:08:56.251 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:56.646 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-13T02:08:56.646 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280538 2026-02-13T02:08:56.646 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280538 2026-02-13T02:08:56.646 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-38654705692 2-60129542172 3-279172874257 4-94489280538' 2026-02-13T02:08:56.646 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:56.646 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-13T02:08:56.646 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=403726925826 2026-02-13T02:08:56.647 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 403726925826 2026-02-13T02:08:56.647 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-38654705692 2-60129542172 3-279172874257 4-94489280538 5-403726925826' 2026-02-13T02:08:56.647 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:56.647 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-13T02:08:56.647 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986201 2026-02-13T02:08:56.647 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986201 2026-02-13T02:08:56.647 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-38654705692 2-60129542172 3-279172874257 4-94489280538 5-403726925826 6-133143986201' 2026-02-13T02:08:56.647 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:56.647 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-13T02:08:56.647 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855384 2026-02-13T02:08:56.647 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855384 2026-02-13T02:08:56.647 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-38654705692 2-60129542172 3-279172874257 4-94489280538 5-403726925826 6-133143986201 7-150323855384' 2026-02-13T02:08:56.648 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:56.648 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-13T02:08:56.648 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691863 2026-02-13T02:08:56.648 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691863 2026-02-13T02:08:56.648 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-38654705692 2-60129542172 3-279172874257 4-94489280538 5-403726925826 6-133143986201 7-150323855384 8-171798691863' 2026-02-13T02:08:56.648 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:56.648 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-13T02:08:56.648 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=403726925826 2026-02-13T02:08:56.648 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 403726925826 2026-02-13T02:08:56.648 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-38654705692 2-60129542172 3-279172874257 4-94489280538 5-403726925826 6-133143986201 7-150323855384 8-171798691863 9-403726925826' 2026-02-13T02:08:56.648 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:56.695 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubunt:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-13T02:08:56.695 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280539 2026-02-13T02:08:56.695 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280539 2026-02-13T02:08:56.695 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-38654705693 2-60129542173 3-279172874258 4-94489280539' 2026-02-13T02:08:56.695 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:56.695 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-13T02:08:56.695 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=403726925827 2026-02-13T02:08:56.695 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 403726925827 2026-02-13T02:08:56.696 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-38654705693 2-60129542173 3-279172874258 4-94489280539 5-403726925827' 2026-02-13T02:08:56.696 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:56.696 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-13T02:08:56.696 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986202 2026-02-13T02:08:56.696 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986202 2026-02-13T02:08:56.696 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-38654705693 2-60129542173 3-279172874258 4-94489280539 5-403726925827 6-133143986202' 2026-02-13T02:08:56.696 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:56.696 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-13T02:08:56.696 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855385 2026-02-13T02:08:56.696 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855385 2026-02-13T02:08:56.696 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-38654705693 2-60129542173 3-279172874258 4-94489280539 5-403726925827 6-133143986202 7-150323855385' 2026-02-13T02:08:56.696 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:56.697 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-13T02:08:56.697 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691864 2026-02-13T02:08:56.697 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691864 2026-02-13T02:08:56.697 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-38654705693 2-60129542173 3-279172874258 4-94489280539 5-403726925827 6-133143986202 7-150323855385 8-171798691864' 2026-02-13T02:08:56.697 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:08:56.697 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-13T02:08:56.697 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=403726925827 2026-02-13T02:08:56.697 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 403726925827 2026-02-13T02:08:56.697 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-38654705693 2-60129542173 3-279172874258 4-94489280539 5-403726925827 6-133143986202 7-150323855385 8-171798691864 9-403726925827' 2026-02-13T02:08:56.697 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:57.211 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836510 2026-02-13T02:08:57.212 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:57.212 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:08:57.212 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836510 2026-02-13T02:08:57.212 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:57.212 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836510 2026-02-13T02:08:57.212 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2026-02-13T02:08:57.212 INFO:tasks.workunit.client.0.trial112.stderr:221931: waiting osd.0 seq 21474836510 2026-02-13T02:08:57.212 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:08:57.212 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836510 -lt 21474836510 2026-02-13T02:08:57.212 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:57.213 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705693 2026-02-13T02:08:57.213 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:57.213 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:08:57.213 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705693 2026-02-13T02:08:57.213 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:57.213 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705693 2026-02-13T02:08:57.213 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705693' 2026-02-13T02:08:57.213 INFO:tasks.workunit.client.0.trial112.stderr:221931: waiting osd.1 seq 38654705693 2026-02-13T02:08:57.213 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:08:57.213 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705694 -lt 38654705693 2026-02-13T02:08:57.213 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:57.213 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542173 2026-02-13T02:08:57.213 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:57.214 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:08:57.214 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542173 2026-02-13T02:08:57.214 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:57.214 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542173 2026-02-13T02:08:57.214 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542173' 2026-02-13T02:08:57.214 INFO:tasks.workunit.client.0.trial112.stderr:221931: waiting osd.2 seq 60129542173 2026-02-13T02:08:57.214 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:08:57.214 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542173 -lt 60129542173 2026-02-13T02:08:57.214 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:57.214 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-279172874258 2026-02-13T02:08:57.214 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:57.214 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:08:57.742 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-279172874258 2026-02-13T02:08:57.742 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:57.742 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=279172874258 2026-02-13T02:08:57.742 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 279172874258' 2026-02-13T02:08:57.743 INFO:tasks.workunit.client.0.trial112.stderr:221931: waiting osd.3 seq 279172874258 2026-02-13T02:08:57.743 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:08:57.743 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 279172874259 -lt 279172874258 2026-02-13T02:08:57.743 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:57.743 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-94489280539 2026-02-13T02:08:57.743 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:57.743 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-13T02:08:57.743 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-94489280539 2026-02-13T02:08:57.743 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:57.743 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280539 2026-02-13T02:08:57.743 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 94489280539' 2026-02-13T02:08:57.743 INFO:tasks.workunit.client.0.trial112.stderr:221931: waiting osd.4 seq 94489280539 2026-02-13T02:08:57.744 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-13T02:08:57.744 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280539 -lt 94489280539 2026-02-13T02:08:57.744 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:57.744 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-403726925827 2026-02-13T02:08:57.744 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:57.744 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-13T02:08:57.744 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-403726925827 2026-02-13T02:08:57.744 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:57.744 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=403726925827 2026-02-13T02:08:57.744 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 403726925827' 2026-02-13T02:08:57.744 INFO:tasks.workunit.client.0.trial112.stderr:221931: waiting osd.5 seq 403726925827 2026-02-13T02:08:57.744 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-13T02:08:57.744 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 403726925827 -lt 403726925827 2026-02-13T02:08:57.745 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:57.745 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-133143986202 2026-02-13T02:08:57.745 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:57.745 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-13T02:08:57.745 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-133143986202 2026-02-13T02:08:57.745 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:57.745 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=133143986202 2026-02-13T02:08:57.745 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 133143986202' 2026-02-13T02:08:57.745 INFO:tasks.workunit.client.0.trial112.stderr:221931: waiting osd.6 seq 133143986202 2026-02-13T02:08:58.325 INFO:tasks.workunit.client.0.trial112.stderr:221931: //homu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836509 2026-02-13T02:08:58.325 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:58.325 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:08:58.326 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836509 2026-02-13T02:08:58.326 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:58.326 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836509 2026-02-13T02:08:58.326 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2026-02-13T02:08:58.326 INFO:tasks.workunit.client.0.trial112.stderr:221933: waiting osd.0 seq 21474836509 2026-02-13T02:08:58.326 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:08:58.326 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836508 -lt 21474836509 2026-02-13T02:08:58.326 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:08:58.326 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:08:58.326 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:08:58.326 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836510 -lt 21474836509 2026-02-13T02:08:58.327 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:58.327 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705692 2026-02-13T02:08:58.327 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:58.327 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:08:58.327 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705692 2026-02-13T02:08:58.327 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:58.327 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705692 2026-02-13T02:08:58.327 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705692' 2026-02-13T02:08:58.327 INFO:tasks.workunit.client.0.trial112.stderr:221933: waiting osd.1 seq 38654705692 2026-02-13T02:08:58.327 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:08:58.327 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705694 -lt 38654705692 2026-02-13T02:08:58.327 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:58.327 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542172 2026-02-13T02:08:58.328 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:58.328 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:08:58.328 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542172 2026-02-13T02:08:58.328 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:58.328 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542172 2026-02-13T02:08:58.328 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542172' 2026-02-13T02:08:58.328 INFO:tasks.workunit.client.0.trial112.stderr:221933: waiting osd.2 seq 60129542172 2026-02-13T02:08:58.328 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:08:58.328 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542173 -lt 60129542172 2026-02-13T02:08:58.409 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pge/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-13T02:08:58.410 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 133143986202 -lt 133143986202 2026-02-13T02:08:58.410 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:58.410 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-150323855385 2026-02-13T02:08:58.410 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:58.410 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-13T02:08:58.410 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-150323855385 2026-02-13T02:08:58.410 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:58.410 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855385 2026-02-13T02:08:58.411 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 150323855385' 2026-02-13T02:08:58.411 INFO:tasks.workunit.client.0.trial112.stderr:221931: waiting osd.7 seq 150323855385 2026-02-13T02:08:58.411 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-13T02:08:58.411 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855385 -lt 150323855385 2026-02-13T02:08:58.411 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:58.411 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-171798691864 2026-02-13T02:08:58.411 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:58.411 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-13T02:08:58.411 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-171798691864 2026-02-13T02:08:58.411 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:58.411 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691864 2026-02-13T02:08:58.411 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 171798691864' 2026-02-13T02:08:58.411 INFO:tasks.workunit.client.0.trial112.stderr:221931: waiting osd.8 seq 171798691864 2026-02-13T02:08:58.412 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-13T02:08:58.412 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691864 -lt 171798691864 2026-02-13T02:08:58.412 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:58.412 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-403726925827 2026-02-13T02:08:58.412 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:58.412 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-13T02:08:58.412 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-403726925827 2026-02-13T02:08:58.412 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:58.412 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=403726925827 2026-02-13T02:08:58.412 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 403726925827' 2026-02-13T02:08:58.412 INFO:tasks.workunit.client.0.trial112.stderr:221931: waiting osd.9 seq 403726925827 2026-02-13T02:08:58.413 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-13T02:08:58.413 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 403726925827 -lt 403726925827 2026-02-13T02:08:58.413 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:08:58.856 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalo_stats: for s in $seqs 2026-02-13T02:08:58.856 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-279172874257 2026-02-13T02:08:58.856 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:58.857 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:08:58.857 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-279172874257 2026-02-13T02:08:58.857 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:58.857 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=279172874257 2026-02-13T02:08:58.857 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 279172874257' 2026-02-13T02:08:58.857 INFO:tasks.workunit.client.0.trial112.stderr:221933: waiting osd.3 seq 279172874257 2026-02-13T02:08:58.857 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:08:58.857 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 279172874259 -lt 279172874257 2026-02-13T02:08:58.857 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:58.857 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-94489280538 2026-02-13T02:08:58.857 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:58.857 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-13T02:08:58.858 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-94489280538 2026-02-13T02:08:58.858 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:58.858 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280538 2026-02-13T02:08:58.858 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 94489280538' 2026-02-13T02:08:58.858 INFO:tasks.workunit.client.0.trial112.stderr:221933: waiting osd.4 seq 94489280538 2026-02-13T02:08:58.858 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-13T02:08:58.858 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280539 -lt 94489280538 2026-02-13T02:08:58.858 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:58.858 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-403726925826 2026-02-13T02:08:58.858 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:58.858 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-13T02:08:58.858 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-403726925826 2026-02-13T02:08:58.858 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:58.859 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=403726925826 2026-02-13T02:08:58.859 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 403726925826' 2026-02-13T02:08:58.859 INFO:tasks.workunit.client.0.trial112.stderr:221933: waiting osd.5 seq 403726925826 2026-02-13T02:08:58.859 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-13T02:08:58.859 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 403726925827 -lt 403726925826 2026-02-13T02:08:58.859 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:58.859 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-133143986201 2026-02-13T02:08:58.859 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:58.859 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-13T02:08:58.971 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalonene/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:08:58.971 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:08:58.972 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:08:58.972 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:08:58.972 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:08:58.972 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:08:58.972 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:08:58.972 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:08:58.972 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:08:58.972 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:08:58.972 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:08:58.972 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:08:58.972 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:08:58.972 INFO:tasks.workunit.client.0.trial112.stderr:221931: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:08:58.973 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:08:58.973 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:08:58.973 INFO:tasks.workunit.client.0.trial112.stderr:221931: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:08:58.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-13T02:08:58.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-13T02:08:58.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 221929 2026-02-13T02:08:59.380 INFO:tasks.workunit.client.0.trial112.stderr:/ceph-helpers.sh:2268: flush_pg_stats: echo 6-133143986201 2026-02-13T02:08:59.380 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:59.381 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=133143986201 2026-02-13T02:08:59.381 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 133143986201' 2026-02-13T02:08:59.381 INFO:tasks.workunit.client.0.trial112.stderr:221933: waiting osd.6 seq 133143986201 2026-02-13T02:08:59.381 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-13T02:08:59.381 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 133143986202 -lt 133143986201 2026-02-13T02:08:59.381 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:59.381 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-150323855384 2026-02-13T02:08:59.381 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:59.381 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-13T02:08:59.381 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-150323855384 2026-02-13T02:08:59.382 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:59.382 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855384 2026-02-13T02:08:59.382 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 150323855384' 2026-02-13T02:08:59.382 INFO:tasks.workunit.client.0.trial112.stderr:221933: waiting osd.7 seq 150323855384 2026-02-13T02:08:59.382 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-13T02:08:59.382 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855386 -lt 150323855384 2026-02-13T02:08:59.382 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:59.382 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-171798691863 2026-02-13T02:08:59.382 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:59.382 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-13T02:08:59.382 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-171798691863 2026-02-13T02:08:59.382 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:59.382 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691863 2026-02-13T02:08:59.383 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 171798691863' 2026-02-13T02:08:59.383 INFO:tasks.workunit.client.0.trial112.stderr:221933: waiting osd.8 seq 171798691863 2026-02-13T02:08:59.383 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-13T02:08:59.383 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691865 -lt 171798691863 2026-02-13T02:08:59.383 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:08:59.383 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-403726925826 2026-02-13T02:08:59.383 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:08:59.383 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-13T02:08:59.383 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-403726925826 2026-02-13T02:08:59.383 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:08:59.383 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=403726925826 2026-02-13T02:09:00.107 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 403726925826' 2026-02-13T02:09:00.107 INFO:tasks.workunit.client.0.trial112.stderr:221933: waiting osd.9 seq 403726925826 2026-02-13T02:09:00.107 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-13T02:09:00.107 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 403726925827 -lt 403726925826 2026-02-13T02:09:00.107 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:09:00.108 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:09:00.108 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:09:00.108 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:09:00.108 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:09:00.108 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:09:00.108 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:09:00.108 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:09:00.108 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:09:00.108 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:09:00.108 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:09:00.108 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:09:00.108 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:09:00.109 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:09:00.109 INFO:tasks.workunit.client.0.trial112.stderr:221933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:09:00.109 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:09:00.109 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:09:00.109 INFO:tasks.workunit.client.0.trial112.stderr:221933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:09:00.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-13T02:09:00.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-13T02:09:00.110 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-13T02:09:00.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-13T02:09:00.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:272: corrupt_and_repair_two: return_code=0 2026-02-13T02:09:00.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-13T02:09:00.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:275: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-13T02:09:00.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:276: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-13T02:09:00.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:323: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2026-02-13T02:09:00.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:245: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2026-02-13T02:09:00.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:246: corrupt_and_repair_two: local poolname=ecpool 2026-02-13T02:09:00.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:247: corrupt_and_repair_two: local first=3 2026-02-13T02:09:00.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_two: local second=5 2026-02-13T02:09:00.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_two: pids= 2026-02-13T02:09:00.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-13T02:09:00.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-13T02:09:00.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-13T02:09:00.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 224778"' 2026-02-13T02:09:00.131 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 224778' 2026-02-13T02:09:00.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2026-02-13T02:09:00.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-13T02:09:00.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-13T02:09:00.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 224779"' 2026-02-13T02:09:00.131 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 224779' 2026-02-13T02:09:00.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_two: wait_background pids 2026-02-13T02:09:00.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 224778 224779' 2026-02-13T02:09:00.132 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-13T02:09:00.132 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-13T02:09:00.132 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 224778 2026-02-13T02:09:00.132 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-13T02:09:00.132 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/224781: /' 2026-02-13T02:09:00.132 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2026-02-13T02:09:00.132 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/224783: /' 2026-02-13T02:09:00.734 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:09:00.735 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:09:00.735 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-13T02:09:00.735 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:09:00.735 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2026-02-13T02:09:00.735 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:09:00.735 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:09:00.735 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-13T02:09:00.735 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:09:00.735 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-13T02:09:00.735 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:09:00.735 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:09:00.736 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:09:00.736 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:09:00.736 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:09:00.736 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:09:00.736 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2026-02-13T02:09:00.736 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:09:00.736 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:09:00.736 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-13T02:09:00.736 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:09:00.736 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:09:00.736 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2026-02-13T02:09:00.736 INFO:tasks.workunit.client.0.trial112.stderr:224781: remove 0#2:eb822e21:::SOMETHING:head# 2026-02-13T02:09:00.737 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-13T02:09:00.737 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:09:00.737 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:09:00.737 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-13T02:09:00.737 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:09:00.737 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:09:00.737 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:09:00.737 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:09:00.738 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:09:00.738 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:09:00.738 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-13T02:09:00.738 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-13T02:09:00.738 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:09:00.738 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:09:00.738 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:09:00.739 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:09:00.739 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:09:00.739 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:09:00.739 INFO:tasks.workunit.client.0.trial112.stderr:224781: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:09:00.739 INFO:tasks.workunit.client.0.trial112.stderr:224781: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:09:00.739 INFO:tasks.workunit.client.0.trial112.stderr:224781: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:09:00.739 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:09:00.739 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:09:00.739 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:09:00.739 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:09:00.739 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:09:00.739 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:09:00.740 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:09:00.740 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:09:00.740 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:09:00.740 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:09:00.740 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:09:00.740 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-13T02:09:00.740 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-13T02:09:00.740 INFO:tasks.workunit.client.0.trial112.stderr:224781: start osd.3 2026-02-13T02:09:00.740 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:09:00.750 INFO:tasks.workunit.client.0.trial112.stderr:2224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:09:00.751 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:09:00.751 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2026-02-13T02:09:00.751 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:09:00.751 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2026-02-13T02:09:00.751 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:09:00.751 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:09:00.751 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2026-02-13T02:09:00.751 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:09:00.751 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2026-02-13T02:09:00.751 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:09:00.752 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:09:00.752 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:09:00.752 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:09:00.752 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:09:00.752 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:09:00.752 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2026-02-13T02:09:00.752 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:09:00.752 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:09:00.752 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2026-02-13T02:09:00.752 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:09:00.752 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2026-02-13T02:09:00.752 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2026-02-13T02:09:00.752 INFO:tasks.workunit.client.0.trial112.stderr:224783: remove 1#2:eb822e21:::SOMETHING:head# 2026-02-13T02:09:00.753 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2026-02-13T02:09:00.753 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:09:00.753 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:09:00.753 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2026-02-13T02:09:00.753 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:09:00.753 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2026-02-13T02:09:00.753 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:09:00.753 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:09:00.754 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:09:00.754 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:09:00.754 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2026-02-13T02:09:00.754 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2026-02-13T02:09:00.754 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:09:00.755 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:09:00.755 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:09:00.755 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:09:00.755 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:09:00.755 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:09:00.755 INFO:tasks.workunit.client.0.trial112.stderr:224783: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:09:00.755 INFO:tasks.workunit.client.0.trial112.stderr:224783: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:09:00.755 INFO:tasks.workunit.client.0.trial112.stderr:224783: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:09:00.755 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:09:00.755 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:09:00.755 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:09:00.755 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:09:00.756 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:09:00.756 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:09:00.756 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:09:00.756 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:09:00.756 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:09:00.756 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:09:00.756 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:09:00.756 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2026-02-13T02:09:00.756 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2026-02-13T02:09:00.756 INFO:tasks.workunit.client.0.trial112.stderr:224783: start osd.5 2026-02-13T02:09:00.756 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:09:02.240 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-13T02:09:02.240 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-13T02:09:02.240 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:09:02.240 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:09:02.240 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:09:02.240 INFO:tasks.workunit.client.0.trial112.stderr:224781: 2026-02-13T02:09:00.750+0000 7faaef750a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:09:02.240 INFO:tasks.workunit.client.0.trial112.stderr:224781: 2026-02-13T02:09:00.750+0000 7faaef750a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:09:02.240 INFO:tasks.workunit.client.0.trial112.stderr:224781: 2026-02-13T02:09:00.754+0000 7faaef750a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:09:02.240 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-13T02:09:02.241 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:09:02.241 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-13T02:09:02.241 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:09:02.241 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:09:02.241 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:09:02.241 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:09:02.241 INFO:tasks.workunit.client.0.trial112.stderr:224781: 0 2026-02-13T02:09:02.241 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:09:02.241 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:09:02.241 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:09:02.241 INFO:tasks.workunit.client.0.trial112.stderr:224781: 2026-02-13T02:09:01.238+0000 7faaef750a40 -1 Falling back to public interface 2026-02-13T02:09:02.241 INFO:tasks.workunit.client.0.trial112.stderr:224781: 2026-02-13T02:09:01.942+0000 7faaef750a40 -1 osd.3 96 log_to_monitors true 2026-02-13T02:09:02.241 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:09:02.242 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:09:02.242 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:09:02.242 INFO:tasks.workunit.client.0.trial112.stderr:224781: 1 2026-02-13T02:09:02.242 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:09:02.242 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:09:02.242 INFO:tasks.workunit.client.0.trial112.stderr:224781: osd.3 up in weight 1 up_from 101 up_thru 95 down_at 97 last_clean_interval [65,96) [v2:127.0.0.1:6874/3440098291,v1:127.0.0.1:6875/3440098291] [v2:127.0.0.1:6876/3440098291,v1:127.0.0.1:6877/3440098291] exists,up eecbad03-21ed-4f9f-89e2-0664db8d736e 2026-02-13T02:09:02.242 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:09:02.242 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:09:02.242 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:09:02.242 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:09:02.242 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:09:02.242 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:09:02.243 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:09:02.243 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:09:02.243 INFO:tasks.workunit.client.0.trial112.stderr:224781: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:09:02.243 INFO:tasks.workunit.client.0.trial112.stderr:224781: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: 24783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2026-02-13T02:09:02.243 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2026-02-13T02:09:02.243 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:09:02.244 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:09:02.244 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:09:02.244 INFO:tasks.workunit.client.0.trial112.stderr:224783: 2026-02-13T02:09:00.762+0000 7f3d70316a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:09:02.244 INFO:tasks.workunit.client.0.trial112.stderr:224783: 2026-02-13T02:09:00.766+0000 7f3d70316a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:09:02.244 INFO:tasks.workunit.client.0.trial112.stderr:224783: 2026-02-13T02:09:00.766+0000 7f3d70316a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:09:02.244 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2026-02-13T02:09:02.244 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:09:02.244 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-13T02:09:02.244 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:09:02.244 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:09:02.244 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:09:02.244 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:09:02.245 INFO:tasks.workunit.client.0.trial112.stderr:224783: 0 2026-02-13T02:09:02.245 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:09:02.245 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-13T02:09:02.245 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:09:02.245 INFO:tasks.workunit.client.0.trial112.stderr:224783: 2026-02-13T02:09:01.198+0000 7f3d70316a40 -1 Falling back to public interface 2026-02-13T02:09:02.245 INFO:tasks.workunit.client.0.trial112.stderr:224783: 2026-02-13T02:09:01.866+0000 7f3d70316a40 -1 osd.5 96 log_to_monitors true 2026-02-13T02:09:02.245 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:09:02.245 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:09:02.245 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:09:02.245 INFO:tasks.workunit.client.0.trial112.stderr:224783: 1 2026-02-13T02:09:02.245 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:09:02.245 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-13T02:09:02.245 INFO:tasks.workunit.client.0.trial112.stderr:224783: osd.5 up in weight 1 up_from 100 up_thru 100 down_at 97 last_clean_interval [94,96) [v2:127.0.0.1:6826/1981369699,v1:127.0.0.1:6827/1981369699] [v2:127.0.0.1:6828/1981369699,v1:127.0.0.1:6829/1981369699] exists,up 03eef3ce-a0e3-4fec-86e3-c40fb660505f 2026-02-13T02:09:02.246 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:09:02.246 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:09:02.246 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:09:02.246 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:09:02.246 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:09:02.246 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:09:02.246 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:09:02.246 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:09:02.246 INFO:tasks.workunit.client.0.trial112.stderr:224783: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:09:02.807 INFO:tasks.workunit.client.0.trial112.stderr:224783: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593:get_timeout_delays: echo true 2026-02-13T02:09:02.807 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:09:02.807 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:09:02.808 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:09:02.808 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:09:02.808 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:09:02.808 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:09:02.808 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:09:02.808 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:09:02.808 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:09:02.808 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:09:02.808 INFO:tasks.workunit.client.0.trial112.stderr:224781: 1 2026-02-13T02:09:02.808 INFO:tasks.workunit.client.0.trial112.stderr:224781: 2 2026-02-13T02:09:02.808 INFO:tasks.workunit.client.0.trial112.stderr:224781: 3 2026-02-13T02:09:02.808 INFO:tasks.workunit.client.0.trial112.stderr:224781: 4 2026-02-13T02:09:02.809 INFO:tasks.workunit.client.0.trial112.stderr:224781: 5 2026-02-13T02:09:02.809 INFO:tasks.workunit.client.0.trial112.stderr:224781: 6 2026-02-13T02:09:02.809 INFO:tasks.workunit.client.0.trial112.stderr:224781: 7 2026-02-13T02:09:02.809 INFO:tasks.workunit.client.0.trial112.stderr:224781: 8 2026-02-13T02:09:02.809 INFO:tasks.workunit.client.0.trial112.stderr:224781: 9' 2026-02-13T02:09:02.809 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:09:02.809 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:02.809 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:09:02.809 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836513 2026-02-13T02:09:02.809 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836513 2026-02-13T02:09:02.809 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513' 2026-02-13T02:09:02.810 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:02.810 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:09:02.810 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705696 2026-02-13T02:09:02.810 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705696 2026-02-13T02:09:02.810 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-38654705696' 2026-02-13T02:09:02.810 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:02.810 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:09:02.810 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542175 2026-02-13T02:09:02.810 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542175 2026-02-13T02:09:02.810 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-38654705696 2-60129542175' 2026-02-13T02:09:02.810 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:02.810 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:09:02.810 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=433791696898 2026-02-13T02:09:02.811 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 433791696898 2026-02-13T02:09:02.811 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-38654705696 2-60129542175 3-433791696898' 2026-02-13T02:09:02.811 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:02.811 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258 get_timeout_delays: echo true 2026-02-13T02:09:02.811 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:09:02.811 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:09:02.812 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:09:02.812 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:09:02.812 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:09:02.812 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:09:02.812 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:09:02.812 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:09:02.812 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:09:02.812 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:09:02.812 INFO:tasks.workunit.client.0.trial112.stderr:224783: 1 2026-02-13T02:09:02.812 INFO:tasks.workunit.client.0.trial112.stderr:224783: 2 2026-02-13T02:09:02.812 INFO:tasks.workunit.client.0.trial112.stderr:224783: 3 2026-02-13T02:09:02.812 INFO:tasks.workunit.client.0.trial112.stderr:224783: 4 2026-02-13T02:09:02.812 INFO:tasks.workunit.client.0.trial112.stderr:224783: 5 2026-02-13T02:09:02.813 INFO:tasks.workunit.client.0.trial112.stderr:224783: 6 2026-02-13T02:09:02.813 INFO:tasks.workunit.client.0.trial112.stderr:224783: 7 2026-02-13T02:09:02.813 INFO:tasks.workunit.client.0.trial112.stderr:224783: 8 2026-02-13T02:09:02.813 INFO:tasks.workunit.client.0.trial112.stderr:224783: 9' 2026-02-13T02:09:02.813 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:09:02.813 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:02.813 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:09:02.813 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836512 2026-02-13T02:09:02.813 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836512 2026-02-13T02:09:02.813 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512' 2026-02-13T02:09:02.813 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:02.813 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:09:02.814 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705697 2026-02-13T02:09:02.814 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705697 2026-02-13T02:09:02.814 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512 1-38654705697' 2026-02-13T02:09:02.814 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:02.814 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:09:02.814 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542176 2026-02-13T02:09:02.814 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542176 2026-02-13T02:09:02.814 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512 1-38654705697 2-60129542176' 2026-02-13T02:09:02.814 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:02.814 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:09:02.814 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=433791696899 2026-02-13T02:09:02.814 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 433791696899 2026-02-13T02:09:02.814 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512 1-38654705697 2-60129542176 3-433791696899' 2026-02-13T02:09:02.815 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:03.256 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:225: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-13T02:09:03.256 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280542 2026-02-13T02:09:03.256 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280542 2026-02-13T02:09:03.256 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-38654705696 2-60129542175 3-433791696898 4-94489280542' 2026-02-13T02:09:03.256 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:03.256 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-13T02:09:03.257 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=429496729603 2026-02-13T02:09:03.257 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 429496729603 2026-02-13T02:09:03.257 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-38654705696 2-60129542175 3-433791696898 4-94489280542 5-429496729603' 2026-02-13T02:09:03.257 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:03.257 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-13T02:09:03.257 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986204 2026-02-13T02:09:03.257 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986204 2026-02-13T02:09:03.257 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-38654705696 2-60129542175 3-433791696898 4-94489280542 5-429496729603 6-133143986204' 2026-02-13T02:09:03.257 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:03.257 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-13T02:09:03.257 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855387 2026-02-13T02:09:03.257 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855387 2026-02-13T02:09:03.258 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-38654705696 2-60129542175 3-433791696898 4-94489280542 5-429496729603 6-133143986204 7-150323855387' 2026-02-13T02:09:03.258 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:03.258 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-13T02:09:03.258 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691867 2026-02-13T02:09:03.258 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691867 2026-02-13T02:09:03.258 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-38654705696 2-60129542175 3-433791696898 4-94489280542 5-429496729603 6-133143986204 7-150323855387 8-171798691867' 2026-02-13T02:09:03.258 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:03.258 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-13T02:09:03.258 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=403726925829 2026-02-13T02:09:03.258 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 403726925829 2026-02-13T02:09:03.258 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-38654705696 2-60129542175 3-433791696898 4-94489280542 5-429496729603 6-133143986204 7-150323855387 8-171798691867 9-403726925829' 2026-02-13T02:09:03.258 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:03.272 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/ce8: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-13T02:09:03.272 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280543 2026-02-13T02:09:03.272 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280543 2026-02-13T02:09:03.272 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512 1-38654705697 2-60129542176 3-433791696899 4-94489280543' 2026-02-13T02:09:03.272 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:03.272 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-13T02:09:03.272 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=429496729602 2026-02-13T02:09:03.272 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 429496729602 2026-02-13T02:09:03.272 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512 1-38654705697 2-60129542176 3-433791696899 4-94489280543 5-429496729602' 2026-02-13T02:09:03.272 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:03.272 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-13T02:09:03.273 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986205 2026-02-13T02:09:03.273 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986205 2026-02-13T02:09:03.273 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512 1-38654705697 2-60129542176 3-433791696899 4-94489280543 5-429496729602 6-133143986205' 2026-02-13T02:09:03.273 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:03.273 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-13T02:09:03.273 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855388 2026-02-13T02:09:03.273 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855388 2026-02-13T02:09:03.273 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512 1-38654705697 2-60129542176 3-433791696899 4-94489280543 5-429496729602 6-133143986205 7-150323855388' 2026-02-13T02:09:03.273 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:03.273 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-13T02:09:03.273 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691868 2026-02-13T02:09:03.274 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691868 2026-02-13T02:09:03.274 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512 1-38654705697 2-60129542176 3-433791696899 4-94489280543 5-429496729602 6-133143986205 7-150323855388 8-171798691868' 2026-02-13T02:09:03.274 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:03.274 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-13T02:09:03.274 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=403726925830 2026-02-13T02:09:03.274 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 403726925830 2026-02-13T02:09:03.274 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512 1-38654705697 2-60129542176 3-433791696899 4-94489280543 5-429496729602 6-133143986205 7-150323855388 8-171798691868 9-403726925830' 2026-02-13T02:09:03.274 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:03.768 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cphtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836513 2026-02-13T02:09:03.768 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:03.768 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:09:03.768 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836513 2026-02-13T02:09:03.768 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:03.768 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836513 2026-02-13T02:09:03.768 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2026-02-13T02:09:03.768 INFO:tasks.workunit.client.0.trial112.stderr:224781: waiting osd.0 seq 21474836513 2026-02-13T02:09:03.769 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:09:03.769 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836513 -lt 21474836513 2026-02-13T02:09:03.769 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:03.769 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705696 2026-02-13T02:09:03.769 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:03.769 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:09:03.769 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705696 2026-02-13T02:09:03.769 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:03.769 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705696 2026-02-13T02:09:03.769 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705696' 2026-02-13T02:09:03.769 INFO:tasks.workunit.client.0.trial112.stderr:224781: waiting osd.1 seq 38654705696 2026-02-13T02:09:03.769 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:09:03.770 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705697 -lt 38654705696 2026-02-13T02:09:03.770 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:03.770 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542175 2026-02-13T02:09:03.770 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:03.770 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:09:03.770 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542175 2026-02-13T02:09:03.770 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:03.770 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542175 2026-02-13T02:09:03.770 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542175' 2026-02-13T02:09:03.770 INFO:tasks.workunit.client.0.trial112.stderr:224781: waiting osd.2 seq 60129542175 2026-02-13T02:09:03.770 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:09:03.770 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542176 -lt 60129542175 2026-02-13T02:09:03.770 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:03.771 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-433791696898 2026-02-13T02:09:03.771 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:03.771 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:09:03.771 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-433791696898 2026-02-13T02:09:03.775 INFO:tasks.workunit.client.0.trial112.stderr:22ephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836512 2026-02-13T02:09:03.775 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:03.775 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:09:03.775 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836512 2026-02-13T02:09:03.775 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:03.775 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836512 2026-02-13T02:09:03.775 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2026-02-13T02:09:03.776 INFO:tasks.workunit.client.0.trial112.stderr:224783: waiting osd.0 seq 21474836512 2026-02-13T02:09:03.776 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:09:03.776 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836513 -lt 21474836512 2026-02-13T02:09:03.776 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:03.776 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705697 2026-02-13T02:09:03.776 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:03.776 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:09:03.776 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705697 2026-02-13T02:09:03.776 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:03.776 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705697 2026-02-13T02:09:03.776 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705697' 2026-02-13T02:09:03.776 INFO:tasks.workunit.client.0.trial112.stderr:224783: waiting osd.1 seq 38654705697 2026-02-13T02:09:03.776 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:09:03.777 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705697 -lt 38654705697 2026-02-13T02:09:03.777 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:03.777 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542176 2026-02-13T02:09:03.777 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:03.777 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:09:03.777 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542176 2026-02-13T02:09:03.777 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:03.777 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542176 2026-02-13T02:09:03.777 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542176' 2026-02-13T02:09:03.777 INFO:tasks.workunit.client.0.trial112.stderr:224783: waiting osd.2 seq 60129542176 2026-02-13T02:09:03.777 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:09:03.777 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542176 -lt 60129542176 2026-02-13T02:09:03.777 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:03.778 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-433791696899 2026-02-13T02:09:03.778 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:03.778 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:09:03.778 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-433791696899 2026-02-13T02:09:05.432 INFO:tasks.workunit.client.0.trial112.stderr:24781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:05.432 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=433791696898 2026-02-13T02:09:05.432 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 433791696898' 2026-02-13T02:09:05.432 INFO:tasks.workunit.client.0.trial112.stderr:224781: waiting osd.3 seq 433791696898 2026-02-13T02:09:05.432 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:09:05.432 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 433791696898 2026-02-13T02:09:05.433 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:09:05.433 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:09:05.433 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:09:05.433 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 433791696899 -lt 433791696898 2026-02-13T02:09:05.433 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:05.433 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-94489280542 2026-02-13T02:09:05.433 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:05.433 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-13T02:09:05.433 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-94489280542 2026-02-13T02:09:05.433 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:05.433 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280542 2026-02-13T02:09:05.433 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 94489280542' 2026-02-13T02:09:05.434 INFO:tasks.workunit.client.0.trial112.stderr:224781: waiting osd.4 seq 94489280542 2026-02-13T02:09:05.434 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-13T02:09:05.434 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280543 -lt 94489280542 2026-02-13T02:09:05.434 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:05.434 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-429496729603 2026-02-13T02:09:05.434 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:05.434 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-13T02:09:05.434 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-429496729603 2026-02-13T02:09:05.434 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:05.434 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=429496729603 2026-02-13T02:09:05.434 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 429496729603' 2026-02-13T02:09:05.434 INFO:tasks.workunit.client.0.trial112.stderr:224781: waiting osd.5 seq 429496729603 2026-02-13T02:09:05.435 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-13T02:09:05.435 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 429496729603 -lt 429496729603 2026-02-13T02:09:05.435 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:05.435 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-133143986204 2026-02-13T02:09:05.435 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:05.435 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-13T02:09:05.436 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers24783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:05.437 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=433791696899 2026-02-13T02:09:05.437 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 433791696899' 2026-02-13T02:09:05.437 INFO:tasks.workunit.client.0.trial112.stderr:224783: waiting osd.3 seq 433791696899 2026-02-13T02:09:05.437 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:09:05.437 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 433791696899 2026-02-13T02:09:05.437 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:09:05.437 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:09:05.437 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:09:05.437 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 433791696899 -lt 433791696899 2026-02-13T02:09:05.437 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:05.438 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-94489280543 2026-02-13T02:09:05.438 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:05.438 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-13T02:09:05.438 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-94489280543 2026-02-13T02:09:05.438 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:05.438 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280543 2026-02-13T02:09:05.438 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 94489280543' 2026-02-13T02:09:05.438 INFO:tasks.workunit.client.0.trial112.stderr:224783: waiting osd.4 seq 94489280543 2026-02-13T02:09:05.438 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-13T02:09:05.438 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280543 -lt 94489280543 2026-02-13T02:09:05.438 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:05.438 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-429496729602 2026-02-13T02:09:05.438 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:05.439 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-13T02:09:05.439 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-429496729602 2026-02-13T02:09:05.439 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:05.439 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=429496729602 2026-02-13T02:09:05.439 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 429496729602' 2026-02-13T02:09:05.439 INFO:tasks.workunit.client.0.trial112.stderr:224783: waiting osd.5 seq 429496729602 2026-02-13T02:09:05.439 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-13T02:09:05.439 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 429496729603 -lt 429496729602 2026-02-13T02:09:05.439 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:05.439 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-133143986205 2026-02-13T02:09:05.439 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:05.439 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-13T02:09:05.939 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper.sh:2268: flush_pg_stats: echo 6-133143986204 2026-02-13T02:09:05.939 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:05.940 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=133143986204 2026-02-13T02:09:05.940 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 133143986204' 2026-02-13T02:09:05.940 INFO:tasks.workunit.client.0.trial112.stderr:224781: waiting osd.6 seq 133143986204 2026-02-13T02:09:05.940 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-13T02:09:05.940 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 133143986205 -lt 133143986204 2026-02-13T02:09:05.940 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:05.940 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-150323855387 2026-02-13T02:09:05.940 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:05.940 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-13T02:09:05.940 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-150323855387 2026-02-13T02:09:05.941 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:05.941 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855387 2026-02-13T02:09:05.941 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 150323855387' 2026-02-13T02:09:05.941 INFO:tasks.workunit.client.0.trial112.stderr:224781: waiting osd.7 seq 150323855387 2026-02-13T02:09:05.941 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-13T02:09:05.941 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855389 -lt 150323855387 2026-02-13T02:09:05.941 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:05.941 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-171798691867 2026-02-13T02:09:05.941 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:05.941 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-13T02:09:05.941 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-171798691867 2026-02-13T02:09:05.941 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:05.941 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691867 2026-02-13T02:09:05.942 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 171798691867' 2026-02-13T02:09:05.942 INFO:tasks.workunit.client.0.trial112.stderr:224781: waiting osd.8 seq 171798691867 2026-02-13T02:09:05.942 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-13T02:09:05.942 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691868 -lt 171798691867 2026-02-13T02:09:05.942 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:05.942 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-403726925829 2026-02-13T02:09:05.942 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:05.942 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-13T02:09:05.942 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-403726925829 2026-02-13T02:09:05.942 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:05.943 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=403726925829 2026-02-13T02:09:05.943 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 403726925829' 2026-02-13T02:09:05.952 INFO:tasks.workunit.client.0.trial112.stderr:2s.sh:2268: flush_pg_stats: echo 6-133143986205 2026-02-13T02:09:05.952 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:05.952 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=133143986205 2026-02-13T02:09:05.952 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 133143986205' 2026-02-13T02:09:05.953 INFO:tasks.workunit.client.0.trial112.stderr:224783: waiting osd.6 seq 133143986205 2026-02-13T02:09:05.953 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-13T02:09:05.953 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 133143986205 -lt 133143986205 2026-02-13T02:09:05.953 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:05.953 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-150323855388 2026-02-13T02:09:05.953 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:05.953 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-13T02:09:05.953 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-150323855388 2026-02-13T02:09:05.953 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:05.953 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855388 2026-02-13T02:09:05.953 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 150323855388' 2026-02-13T02:09:05.953 INFO:tasks.workunit.client.0.trial112.stderr:224783: waiting osd.7 seq 150323855388 2026-02-13T02:09:05.954 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-13T02:09:05.954 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855389 -lt 150323855388 2026-02-13T02:09:05.954 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:05.954 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-171798691868 2026-02-13T02:09:05.954 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:05.954 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-13T02:09:05.954 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-171798691868 2026-02-13T02:09:05.954 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:05.954 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691868 2026-02-13T02:09:05.954 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 171798691868' 2026-02-13T02:09:05.954 INFO:tasks.workunit.client.0.trial112.stderr:224783: waiting osd.8 seq 171798691868 2026-02-13T02:09:05.954 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-13T02:09:05.954 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691868 -lt 171798691868 2026-02-13T02:09:05.955 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:05.955 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-403726925830 2026-02-13T02:09:05.955 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:05.955 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-13T02:09:05.955 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-403726925830 2026-02-13T02:09:05.955 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:05.955 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=403726925830 2026-02-13T02:09:05.955 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 403726925830' 2026-02-13T02:09:06.653 INFO:tasks.workunit.client.0.trial112.stderr:24781: waiting osd.9 seq 403726925829 2026-02-13T02:09:06.653 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-13T02:09:06.653 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 403726925830 -lt 403726925829 2026-02-13T02:09:06.654 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:09:06.654 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:09:06.654 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:09:06.654 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:09:06.654 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:09:06.654 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:09:06.654 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:09:06.654 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:09:06.654 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:09:06.654 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:09:06.654 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:09:06.655 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:09:06.655 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:09:06.655 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:09:06.655 INFO:tasks.workunit.client.0.trial112.stderr:224781: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:09:06.655 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:09:06.655 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:09:06.655 INFO:tasks.workunit.client.0.trial112.stderr:224781: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:09:06.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-13T02:09:06.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-13T02:09:06.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 224779 2026-02-13T02:09:06.697 INFO:tasks.workunit.client.0.trial112.stderr:224783: waiting osd.9 seq 403726925830 2026-02-13T02:09:06.697 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-13T02:09:06.697 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 403726925830 -lt 403726925830 2026-02-13T02:09:06.697 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:09:06.697 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:09:06.698 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:09:06.698 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:09:06.698 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:09:06.698 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:09:06.698 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:09:06.698 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:09:06.698 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:09:06.698 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:09:06.698 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:09:06.698 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:09:06.698 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:09:06.698 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:09:06.699 INFO:tasks.workunit.client.0.trial112.stderr:224783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:09:06.699 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:09:06.699 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:09:06.699 INFO:tasks.workunit.client.0.trial112.stderr:224783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:09:06.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-13T02:09:06.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-13T02:09:06.700 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-13T02:09:06.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-13T02:09:06.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:257: corrupt_and_repair_two: return_code=0 2026-02-13T02:09:06.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:258: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-13T02:09:06.701 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: get_pg ecpool SOMETHING 2026-02-13T02:09:06.701 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-13T02:09:06.701 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-13T02:09:06.701 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-13T02:09:06.701 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T02:09:06.866 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: local pg=2.0 2026-02-13T02:09:06.866 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:264: corrupt_and_repair_two: repair 2.0 2026-02-13T02:09:06.866 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-13T02:09:06.866 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-13T02:09:06.866 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:09:06.866 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:09:06.867 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:09:06.867 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:09:07.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-13T02:08:47.347459+0000 2026-02-13T02:09:07.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-13T02:09:07.181 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-13T02:09:07.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-13T02:08:47.347459+0000 2026-02-13T02:09:07.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-13T02:09:07.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:08:47.347459+0000 2026-02-13T02:09:07.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:09:07.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:09:07.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:09:07.192 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:09:07.192 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:09:07.192 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:09:07.192 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:09:07.192 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:09:07.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:08:47.347459+0000 '>' 2026-02-13T02:08:47.347459+0000 2026-02-13T02:09:07.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:09:08.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:09:08.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:09:08.356 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:09:08.357 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:09:08.357 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:09:08.357 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:09:08.357 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:09:08.516 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:08:47.347459+0000 '>' 2026-02-13T02:08:47.347459+0000 2026-02-13T02:09:08.516 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:09:09.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:09:09.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:09:09.518 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:09:09.518 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:09:09.518 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:09:09.518 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:09:09.518 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:09:09.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:08:47.347459+0000 '>' 2026-02-13T02:08:47.347459+0000 2026-02-13T02:09:09.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:09:10.678 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:09:10.678 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:09:10.679 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:09:10.679 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:09:10.679 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:09:10.679 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:09:10.679 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:09:10.835 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:08:47.347459+0000 '>' 2026-02-13T02:08:47.347459+0000 2026-02-13T02:09:10.835 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:09:11.836 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:09:11.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:09:11.838 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:09:11.838 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:09:11.838 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:09:11.838 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:09:11.838 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:09:11.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:08:47.347459+0000 '>' 2026-02-13T02:08:47.347459+0000 2026-02-13T02:09:11.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:09:12.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:09:12.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:09:12.997 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:09:12.997 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:09:12.997 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:09:12.998 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:09:12.998 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:09:13.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:09:07.920869+0000 '>' 2026-02-13T02:08:47.347459+0000 2026-02-13T02:09:13.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:09:13.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: corrupt_and_repair_two: pids= 2026-02-13T02:09:13.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-13T02:09:13.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-13T02:09:13.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-13T02:09:13.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 228215"' 2026-02-13T02:09:13.157 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 228215' 2026-02-13T02:09:13.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-13T02:09:13.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-13T02:09:13.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-13T02:09:13.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 228217"' 2026-02-13T02:09:13.158 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 228217' 2026-02-13T02:09:13.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-13T02:09:13.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:271: corrupt_and_repair_two: wait_background pids 2026-02-13T02:09:13.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 228215 228217' 2026-02-13T02:09:13.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-13T02:09:13.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-13T02:09:13.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 228215 2026-02-13T02:09:13.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/228218: /' 2026-02-13T02:09:13.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-13T02:09:13.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/228220: /' 2026-02-13T02:09:13.646 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:09:13.646 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:09:13.646 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-13T02:09:13.646 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:09:13.647 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-13T02:09:13.647 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:09:13.647 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:09:13.647 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-13T02:09:13.647 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:09:13.647 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-13T02:09:13.647 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:09:13.647 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:09:13.647 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:09:13.647 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:09:13.647 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:09:13.647 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:09:13.648 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-13T02:09:13.648 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:09:13.648 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:09:13.648 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-13T02:09:13.648 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:09:13.648 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:09:13.648 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2026-02-13T02:09:13.648 INFO:tasks.workunit.client.0.trial112.stderr:228218: Error getting attr on : 2.0s0_head,0#-4:00000000:::deep_scrub_2.0s0:head#, (61) No data available 2026-02-13T02:09:13.648 INFO:tasks.workunit.client.0.trial112.stderr:228218: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2026-02-13T02:09:13.648 INFO:tasks.workunit.client.0.trial112.stderr:228218: _ 2026-02-13T02:09:13.648 INFO:tasks.workunit.client.0.trial112.stderr:228218: hinfo_key 2026-02-13T02:09:13.648 INFO:tasks.workunit.client.0.trial112.stderr:228218: snapset 2026-02-13T02:09:13.649 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-13T02:09:13.649 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:09:13.649 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:09:13.649 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-13T02:09:13.649 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:09:13.649 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:09:13.649 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:09:13.650 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:09:13.650 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:09:13.650 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:09:13.650 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-13T02:09:13.650 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-13T02:09:13.650 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:09:13.650 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:09:13.650 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:09:13.650 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:09:13.650 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:09:13.651 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:09:13.651 INFO:tasks.workunit.client.0.trial112.stderr:228218: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:09:13.651 INFO:tasks.workunit.client.0.trial112.stderr:228218: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:09:13.651 INFO:tasks.workunit.client.0.trial112.stderr:228218: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:09:13.651 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:09:13.651 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:09:13.651 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:09:13.651 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:09:13.651 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:09:13.651 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:09:13.651 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:09:13.651 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:09:13.651 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:09:13.652 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:09:13.652 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:09:13.652 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-13T02:09:13.652 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-13T02:09:13.652 INFO:tasks.workunit.client.0.trial112.stderr:228218: start osd.3 2026-02-13T02:09:13.670 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:09:13.670 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:09:13.670 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2026-02-13T02:09:13.671 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:09:13.671 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-13T02:09:13.671 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:09:13.671 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:09:13.671 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2026-02-13T02:09:13.671 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:09:13.671 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2026-02-13T02:09:13.671 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:09:13.671 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:09:13.671 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:09:13.671 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:09:13.671 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:09:13.672 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:09:13.672 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-13T02:09:13.672 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:09:13.672 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:09:13.672 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2026-02-13T02:09:13.672 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:09:13.672 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2026-02-13T02:09:13.672 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2026-02-13T02:09:13.672 INFO:tasks.workunit.client.0.trial112.stderr:228220: _ 2026-02-13T02:09:13.672 INFO:tasks.workunit.client.0.trial112.stderr:228220: hinfo_key 2026-02-13T02:09:13.672 INFO:tasks.workunit.client.0.trial112.stderr:228220: snapset 2026-02-13T02:09:13.672 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2026-02-13T02:09:13.673 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:09:13.673 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:09:13.673 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2026-02-13T02:09:13.673 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:09:13.673 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2026-02-13T02:09:13.673 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:09:13.673 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:09:13.674 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:09:13.674 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:09:13.674 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2026-02-13T02:09:13.674 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2026-02-13T02:09:13.674 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:09:13.674 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:09:13.674 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:09:13.674 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:09:13.674 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:09:13.675 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:09:13.675 INFO:tasks.workunit.client.0.trial112.stderr:228220: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:09:13.675 INFO:tasks.workunit.client.0.trial112.stderr:228220: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:09:13.675 INFO:tasks.workunit.client.0.trial112.stderr:228220: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:09:13.675 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:09:13.675 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:09:13.675 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:09:13.675 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:09:13.675 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:09:13.675 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:09:13.675 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:09:13.675 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:09:13.675 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:09:13.676 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:09:13.676 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:09:13.676 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2026-02-13T02:09:13.676 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2026-02-13T02:09:13.676 INFO:tasks.workunit.client.0.trial112.stderr:228220: start osd.5 2026-02-13T02:09:15.138 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recolog' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:09:15.138 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-13T02:09:15.138 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-13T02:09:15.138 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:09:15.138 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:09:15.138 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:09:15.138 INFO:tasks.workunit.client.0.trial112.stderr:228218: 2026-02-13T02:09:13.658+0000 7fb369e0aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:09:15.139 INFO:tasks.workunit.client.0.trial112.stderr:228218: 2026-02-13T02:09:13.662+0000 7fb369e0aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:09:15.139 INFO:tasks.workunit.client.0.trial112.stderr:228218: 2026-02-13T02:09:13.662+0000 7fb369e0aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:09:15.139 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-13T02:09:15.139 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:09:15.139 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-13T02:09:15.139 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:09:15.139 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:09:15.139 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:09:15.139 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:09:15.139 INFO:tasks.workunit.client.0.trial112.stderr:228218: 0 2026-02-13T02:09:15.139 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:09:15.140 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:09:15.140 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:09:15.140 INFO:tasks.workunit.client.0.trial112.stderr:228218: 2026-02-13T02:09:14.086+0000 7fb369e0aa40 -1 Falling back to public interface 2026-02-13T02:09:15.140 INFO:tasks.workunit.client.0.trial112.stderr:228218: 2026-02-13T02:09:14.758+0000 7fb369e0aa40 -1 osd.3 102 log_to_monitors true 2026-02-13T02:09:15.140 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:09:15.140 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:09:15.140 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:09:15.140 INFO:tasks.workunit.client.0.trial112.stderr:228218: 1 2026-02-13T02:09:15.140 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:09:15.140 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:09:15.140 INFO:tasks.workunit.client.0.trial112.stderr:228218: osd.3 up in weight 1 up_from 106 up_thru 106 down_at 103 last_clean_interval [101,102) [v2:127.0.0.1:6826/1748751053,v1:127.0.0.1:6827/1748751053] [v2:127.0.0.1:6828/1748751053,v1:127.0.0.1:6829/1748751053] exists,up eecbad03-21ed-4f9f-89e2-0664db8d736e 2026-02-13T02:09:15.140 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:09:15.141 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:09:15.141 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:09:15.141 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:09:15.141 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:09:15.141 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:09:15.141 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:09:15.182 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeovery_ops 2026-02-13T02:09:15.182 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2026-02-13T02:09:15.183 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2026-02-13T02:09:15.183 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:09:15.183 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:09:15.183 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:09:15.183 INFO:tasks.workunit.client.0.trial112.stderr:228220: 2026-02-13T02:09:13.678+0000 7f415421aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:09:15.183 INFO:tasks.workunit.client.0.trial112.stderr:228220: 2026-02-13T02:09:13.682+0000 7f415421aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:09:15.183 INFO:tasks.workunit.client.0.trial112.stderr:228220: 2026-02-13T02:09:13.682+0000 7f415421aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:09:15.183 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2026-02-13T02:09:15.183 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:09:15.183 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-13T02:09:15.183 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:09:15.184 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:09:15.184 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:09:15.184 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:09:15.184 INFO:tasks.workunit.client.0.trial112.stderr:228220: 0 2026-02-13T02:09:15.184 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:09:15.184 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-13T02:09:15.184 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:09:15.184 INFO:tasks.workunit.client.0.trial112.stderr:228220: 2026-02-13T02:09:14.134+0000 7f415421aa40 -1 Falling back to public interface 2026-02-13T02:09:15.184 INFO:tasks.workunit.client.0.trial112.stderr:228220: 2026-02-13T02:09:14.810+0000 7f415421aa40 -1 osd.5 102 log_to_monitors true 2026-02-13T02:09:15.184 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:09:15.184 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:09:15.184 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:09:15.185 INFO:tasks.workunit.client.0.trial112.stderr:228220: 1 2026-02-13T02:09:15.185 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:09:15.185 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-13T02:09:15.185 INFO:tasks.workunit.client.0.trial112.stderr:228220: osd.5 up in weight 1 up_from 107 up_thru 107 down_at 103 last_clean_interval [100,102) [v2:127.0.0.1:6874/3588917852,v1:127.0.0.1:6875/3588917852] [v2:127.0.0.1:6876/3588917852,v1:127.0.0.1:6877/3588917852] exists,up 03eef3ce-a0e3-4fec-86e3-c40fb660505f 2026-02-13T02:09:15.185 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:09:15.185 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:09:15.185 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:09:15.185 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:09:15.185 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:09:15.185 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:09:15.185 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:09:15.186 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:09:15.186 INFO:tasks.workunit.client.0.trial112.stderr:228220: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:09:15.694 INFO:tasks.workunit.client.0.trial112.stderr:228220: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-heut_delays 90 .1 2026-02-13T02:09:15.695 INFO:tasks.workunit.client.0.trial112.stderr:228218: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:09:15.695 INFO:tasks.workunit.client.0.trial112.stderr:228218: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:09:15.695 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:09:15.695 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:09:15.695 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:09:15.695 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:09:15.695 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:09:15.695 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:09:15.695 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:09:15.695 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:09:15.695 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:09:15.695 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:09:15.696 INFO:tasks.workunit.client.0.trial112.stderr:228218: 1 2026-02-13T02:09:15.696 INFO:tasks.workunit.client.0.trial112.stderr:228218: 2 2026-02-13T02:09:15.696 INFO:tasks.workunit.client.0.trial112.stderr:228218: 3 2026-02-13T02:09:15.696 INFO:tasks.workunit.client.0.trial112.stderr:228218: 4 2026-02-13T02:09:15.696 INFO:tasks.workunit.client.0.trial112.stderr:228218: 5 2026-02-13T02:09:15.696 INFO:tasks.workunit.client.0.trial112.stderr:228218: 6 2026-02-13T02:09:15.696 INFO:tasks.workunit.client.0.trial112.stderr:228218: 7 2026-02-13T02:09:15.696 INFO:tasks.workunit.client.0.trial112.stderr:228218: 8 2026-02-13T02:09:15.696 INFO:tasks.workunit.client.0.trial112.stderr:228218: 9' 2026-02-13T02:09:15.696 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:09:15.696 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:15.696 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:09:15.697 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836517 2026-02-13T02:09:15.697 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836517 2026-02-13T02:09:15.697 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836517' 2026-02-13T02:09:15.697 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:15.697 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:09:15.697 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705700 2026-02-13T02:09:15.697 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705700 2026-02-13T02:09:15.697 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836517 1-38654705700' 2026-02-13T02:09:15.697 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:15.697 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:09:15.697 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542180 2026-02-13T02:09:15.697 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542180 2026-02-13T02:09:15.697 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836517 1-38654705700 2-60129542180' 2026-02-13T02:09:15.698 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:15.698 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:09:15.698 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=455266533378 2026-02-13T02:09:15.698 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 455266533378 2026-02-13T02:09:15.747 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836517 1-38654705700 2-lpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:09:15.747 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:09:15.747 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:09:15.748 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:09:15.748 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:09:15.748 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:09:15.748 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:09:15.748 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:09:15.748 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:09:15.748 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:09:15.748 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:09:15.748 INFO:tasks.workunit.client.0.trial112.stderr:228220: 1 2026-02-13T02:09:15.748 INFO:tasks.workunit.client.0.trial112.stderr:228220: 2 2026-02-13T02:09:15.748 INFO:tasks.workunit.client.0.trial112.stderr:228220: 3 2026-02-13T02:09:15.748 INFO:tasks.workunit.client.0.trial112.stderr:228220: 4 2026-02-13T02:09:15.749 INFO:tasks.workunit.client.0.trial112.stderr:228220: 5 2026-02-13T02:09:15.749 INFO:tasks.workunit.client.0.trial112.stderr:228220: 6 2026-02-13T02:09:15.749 INFO:tasks.workunit.client.0.trial112.stderr:228220: 7 2026-02-13T02:09:15.749 INFO:tasks.workunit.client.0.trial112.stderr:228220: 8 2026-02-13T02:09:15.749 INFO:tasks.workunit.client.0.trial112.stderr:228220: 9' 2026-02-13T02:09:15.749 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:09:15.749 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:15.749 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:09:15.749 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836518 2026-02-13T02:09:15.749 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836518 2026-02-13T02:09:15.749 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518' 2026-02-13T02:09:15.749 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:15.749 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:09:15.750 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705701 2026-02-13T02:09:15.750 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705701 2026-02-13T02:09:15.750 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-38654705701' 2026-02-13T02:09:15.750 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:15.750 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:09:15.750 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542181 2026-02-13T02:09:15.750 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542181 2026-02-13T02:09:15.750 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-38654705701 2-60129542181' 2026-02-13T02:09:15.750 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:15.750 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:09:15.750 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=455266533379 2026-02-13T02:09:15.750 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 455266533379 2026-02-13T02:09:15.750 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-38654705701 2-60129542181 3-455266533379' 2026-02-13T02:09:15.751 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:16.131 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-60129542180 3-455266533378' 2026-02-13T02:09:16.131 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:16.131 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-13T02:09:16.131 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280546 2026-02-13T02:09:16.131 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280546 2026-02-13T02:09:16.131 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836517 1-38654705700 2-60129542180 3-455266533378 4-94489280546' 2026-02-13T02:09:16.131 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:16.132 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-13T02:09:16.132 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=459561500674 2026-02-13T02:09:16.132 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 459561500674 2026-02-13T02:09:16.132 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836517 1-38654705700 2-60129542180 3-455266533378 4-94489280546 5-459561500674' 2026-02-13T02:09:16.132 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:16.132 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-13T02:09:16.132 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986209 2026-02-13T02:09:16.132 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986209 2026-02-13T02:09:16.132 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836517 1-38654705700 2-60129542180 3-455266533378 4-94489280546 5-459561500674 6-133143986209' 2026-02-13T02:09:16.132 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:16.132 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-13T02:09:16.133 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855392 2026-02-13T02:09:16.133 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855392 2026-02-13T02:09:16.133 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836517 1-38654705700 2-60129542180 3-455266533378 4-94489280546 5-459561500674 6-133143986209 7-150323855392' 2026-02-13T02:09:16.133 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:16.133 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-13T02:09:16.133 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691871 2026-02-13T02:09:16.133 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691871 2026-02-13T02:09:16.133 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836517 1-38654705700 2-60129542180 3-455266533378 4-94489280546 5-459561500674 6-133143986209 7-150323855392 8-171798691871' 2026-02-13T02:09:16.133 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:16.133 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-13T02:09:16.133 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=403726925834 2026-02-13T02:09:16.133 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 403726925834 2026-02-13T02:09:16.200 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836517 1-38654705700 2-60129542180 3-455266533378 4-944helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-13T02:09:16.200 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280547 2026-02-13T02:09:16.200 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280547 2026-02-13T02:09:16.201 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-38654705701 2-60129542181 3-455266533379 4-94489280547' 2026-02-13T02:09:16.201 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:16.201 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-13T02:09:16.201 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=459561500675 2026-02-13T02:09:16.201 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 459561500675 2026-02-13T02:09:16.201 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-38654705701 2-60129542181 3-455266533379 4-94489280547 5-459561500675' 2026-02-13T02:09:16.201 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:16.201 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-13T02:09:16.201 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986210 2026-02-13T02:09:16.201 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986210 2026-02-13T02:09:16.201 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-38654705701 2-60129542181 3-455266533379 4-94489280547 5-459561500675 6-133143986210' 2026-02-13T02:09:16.202 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:16.202 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-13T02:09:16.202 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855393 2026-02-13T02:09:16.202 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855393 2026-02-13T02:09:16.202 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-38654705701 2-60129542181 3-455266533379 4-94489280547 5-459561500675 6-133143986210 7-150323855393' 2026-02-13T02:09:16.202 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:16.202 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-13T02:09:16.202 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691872 2026-02-13T02:09:16.202 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691872 2026-02-13T02:09:16.202 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-38654705701 2-60129542181 3-455266533379 4-94489280547 5-459561500675 6-133143986210 7-150323855393 8-171798691872' 2026-02-13T02:09:16.202 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:16.202 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-13T02:09:16.202 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=403726925835 2026-02-13T02:09:16.203 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 403726925835 2026-02-13T02:09:16.203 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-38654705701 2-60129542181 3-455266533379 4-94489280547 5-459561500675 6-133143986210 7-150323855393 8-171798691872 9-403726925835' 2026-02-13T02:09:16.203 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:17.793 INFO:tasks.workunit.client.0.trial112.stderr:228220: /89280546 5-459561500674 6-133143986209 7-150323855392 8-171798691871 9-403726925834' 2026-02-13T02:09:17.794 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:17.794 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836517 2026-02-13T02:09:17.794 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:17.794 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:09:17.794 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836517 2026-02-13T02:09:17.794 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:17.794 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836517 2026-02-13T02:09:17.794 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836517' 2026-02-13T02:09:17.794 INFO:tasks.workunit.client.0.trial112.stderr:228218: waiting osd.0 seq 21474836517 2026-02-13T02:09:17.795 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:09:17.795 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836516 -lt 21474836517 2026-02-13T02:09:17.795 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:09:17.795 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:09:17.795 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:09:17.795 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836518 -lt 21474836517 2026-02-13T02:09:17.795 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:17.795 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705700 2026-02-13T02:09:17.795 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:17.795 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:09:17.795 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705700 2026-02-13T02:09:17.796 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:17.796 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705700 2026-02-13T02:09:17.796 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705700' 2026-02-13T02:09:17.796 INFO:tasks.workunit.client.0.trial112.stderr:228218: waiting osd.1 seq 38654705700 2026-02-13T02:09:17.796 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:09:17.796 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705702 -lt 38654705700 2026-02-13T02:09:17.796 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:17.796 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542180 2026-02-13T02:09:17.796 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:17.796 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:09:17.796 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542180 2026-02-13T02:09:17.797 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:17.797 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542180 2026-02-13T02:09:17.797 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542180' 2026-02-13T02:09:17.797 INFO:tasks.workunit.client.0.trial112.stderr:228218: waiting osd.2 seq 60129542180 2026-02-13T02:09:17.797 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:09:17.859 INFO:tasks.workunit.client.0.trial112.stderr:22/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836518 2026-02-13T02:09:17.859 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:17.860 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:09:17.860 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836518 2026-02-13T02:09:17.860 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:17.860 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836518 2026-02-13T02:09:17.860 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836518' 2026-02-13T02:09:17.860 INFO:tasks.workunit.client.0.trial112.stderr:228220: waiting osd.0 seq 21474836518 2026-02-13T02:09:17.860 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:09:17.860 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836516 -lt 21474836518 2026-02-13T02:09:17.860 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:09:17.860 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:09:17.860 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:09:17.860 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836518 -lt 21474836518 2026-02-13T02:09:17.861 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:17.861 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705701 2026-02-13T02:09:17.861 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:17.861 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:09:17.861 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705701 2026-02-13T02:09:17.861 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:17.861 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705701 2026-02-13T02:09:17.861 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705701' 2026-02-13T02:09:17.861 INFO:tasks.workunit.client.0.trial112.stderr:228220: waiting osd.1 seq 38654705701 2026-02-13T02:09:17.861 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:09:17.861 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705702 -lt 38654705701 2026-02-13T02:09:17.861 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:17.862 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542181 2026-02-13T02:09:17.862 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:17.862 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:09:17.862 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542181 2026-02-13T02:09:17.862 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:17.862 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542181 2026-02-13T02:09:17.862 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542181' 2026-02-13T02:09:17.862 INFO:tasks.workunit.client.0.trial112.stderr:228220: waiting osd.2 seq 60129542181 2026-02-13T02:09:17.862 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:09:17.862 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542181 -lt 60129542181 2026-02-13T02:09:18.317 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542181 -lt 60129542180 2026-02-13T02:09:18.317 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:18.317 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-455266533378 2026-02-13T02:09:18.318 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:18.318 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:09:18.318 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-455266533378 2026-02-13T02:09:18.318 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:18.318 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=455266533378 2026-02-13T02:09:18.318 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 455266533378' 2026-02-13T02:09:18.318 INFO:tasks.workunit.client.0.trial112.stderr:228218: waiting osd.3 seq 455266533378 2026-02-13T02:09:18.318 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:09:18.318 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 455266533379 -lt 455266533378 2026-02-13T02:09:18.318 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:18.318 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-94489280546 2026-02-13T02:09:18.318 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:18.319 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-13T02:09:18.319 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-94489280546 2026-02-13T02:09:18.319 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:18.319 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280546 2026-02-13T02:09:18.319 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 94489280546' 2026-02-13T02:09:18.319 INFO:tasks.workunit.client.0.trial112.stderr:228218: waiting osd.4 seq 94489280546 2026-02-13T02:09:18.319 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-13T02:09:18.319 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280547 -lt 94489280546 2026-02-13T02:09:18.319 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:18.319 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-459561500674 2026-02-13T02:09:18.319 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:18.319 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-13T02:09:18.320 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-459561500674 2026-02-13T02:09:18.320 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:18.320 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=459561500674 2026-02-13T02:09:18.320 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 459561500674' 2026-02-13T02:09:18.320 INFO:tasks.workunit.client.0.trial112.stderr:228218: waiting osd.5 seq 459561500674 2026-02-13T02:09:18.320 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-13T02:09:18.320 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 459561500675 -lt 459561500674 2026-02-13T02:09:18.320 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:18.320 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-133143986209 2026-02-13T02:09:18.371 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalon6: flush_pg_stats: for s in $seqs 2026-02-13T02:09:18.371 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-455266533379 2026-02-13T02:09:18.371 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:18.371 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:09:18.371 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-455266533379 2026-02-13T02:09:18.371 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:18.371 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=455266533379 2026-02-13T02:09:18.371 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 455266533379' 2026-02-13T02:09:18.371 INFO:tasks.workunit.client.0.trial112.stderr:228220: waiting osd.3 seq 455266533379 2026-02-13T02:09:18.372 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:09:18.372 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 455266533379 -lt 455266533379 2026-02-13T02:09:18.372 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:18.372 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-94489280547 2026-02-13T02:09:18.372 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:18.372 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-13T02:09:18.372 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-94489280547 2026-02-13T02:09:18.372 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:18.372 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280547 2026-02-13T02:09:18.372 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 94489280547' 2026-02-13T02:09:18.372 INFO:tasks.workunit.client.0.trial112.stderr:228220: waiting osd.4 seq 94489280547 2026-02-13T02:09:18.372 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-13T02:09:18.372 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280547 -lt 94489280547 2026-02-13T02:09:18.373 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:18.373 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-459561500675 2026-02-13T02:09:18.373 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:18.373 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-13T02:09:18.373 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-459561500675 2026-02-13T02:09:18.373 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:18.373 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=459561500675 2026-02-13T02:09:18.373 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 459561500675' 2026-02-13T02:09:18.373 INFO:tasks.workunit.client.0.trial112.stderr:228220: waiting osd.5 seq 459561500675 2026-02-13T02:09:18.373 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-13T02:09:18.373 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 459561500675 -lt 459561500675 2026-02-13T02:09:18.373 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:18.373 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-133143986210 2026-02-13T02:09:18.374 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:18.374 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-13T02:09:18.830 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qae/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:18.830 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-13T02:09:18.830 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-133143986209 2026-02-13T02:09:18.831 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:18.831 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=133143986209 2026-02-13T02:09:18.831 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 133143986209' 2026-02-13T02:09:18.831 INFO:tasks.workunit.client.0.trial112.stderr:228218: waiting osd.6 seq 133143986209 2026-02-13T02:09:18.831 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-13T02:09:18.831 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 133143986210 -lt 133143986209 2026-02-13T02:09:18.831 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:18.831 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-150323855392 2026-02-13T02:09:18.831 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:18.831 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-13T02:09:18.831 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-150323855392 2026-02-13T02:09:18.831 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:18.832 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855392 2026-02-13T02:09:18.832 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 150323855392' 2026-02-13T02:09:18.832 INFO:tasks.workunit.client.0.trial112.stderr:228218: waiting osd.7 seq 150323855392 2026-02-13T02:09:18.832 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-13T02:09:18.832 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855393 -lt 150323855392 2026-02-13T02:09:18.832 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:18.832 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-171798691871 2026-02-13T02:09:18.832 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:18.832 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-13T02:09:18.832 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-171798691871 2026-02-13T02:09:18.832 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:18.832 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691871 2026-02-13T02:09:18.833 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 171798691871' 2026-02-13T02:09:18.833 INFO:tasks.workunit.client.0.trial112.stderr:228218: waiting osd.8 seq 171798691871 2026-02-13T02:09:18.833 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-13T02:09:18.833 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691872 -lt 171798691871 2026-02-13T02:09:18.833 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:18.833 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-403726925834 2026-02-13T02:09:18.833 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:18.833 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-13T02:09:18.833 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-403726925834 2026-02-13T02:09:18.833 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:18.877 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu//standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-133143986210 2026-02-13T02:09:18.878 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:18.878 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=133143986210 2026-02-13T02:09:18.878 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 133143986210' 2026-02-13T02:09:18.878 INFO:tasks.workunit.client.0.trial112.stderr:228220: waiting osd.6 seq 133143986210 2026-02-13T02:09:18.878 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-13T02:09:18.878 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 133143986210 -lt 133143986210 2026-02-13T02:09:18.878 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:18.878 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-150323855393 2026-02-13T02:09:18.878 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:18.879 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-13T02:09:18.879 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-150323855393 2026-02-13T02:09:18.879 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:18.879 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855393 2026-02-13T02:09:18.879 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 150323855393' 2026-02-13T02:09:18.879 INFO:tasks.workunit.client.0.trial112.stderr:228220: waiting osd.7 seq 150323855393 2026-02-13T02:09:18.879 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-13T02:09:18.879 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855393 -lt 150323855393 2026-02-13T02:09:18.879 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:18.879 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-171798691872 2026-02-13T02:09:18.879 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:18.879 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-13T02:09:18.879 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-171798691872 2026-02-13T02:09:18.880 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:18.880 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691872 2026-02-13T02:09:18.880 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 171798691872' 2026-02-13T02:09:18.880 INFO:tasks.workunit.client.0.trial112.stderr:228220: waiting osd.8 seq 171798691872 2026-02-13T02:09:18.880 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-13T02:09:18.880 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691873 -lt 171798691872 2026-02-13T02:09:18.880 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:18.880 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-403726925835 2026-02-13T02:09:18.880 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:18.880 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-13T02:09:18.880 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-403726925835 2026-02-13T02:09:18.880 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:18.880 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=403726925835 2026-02-13T02:09:19.588 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting ocephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=403726925834 2026-02-13T02:09:19.588 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 403726925834' 2026-02-13T02:09:19.588 INFO:tasks.workunit.client.0.trial112.stderr:228218: waiting osd.9 seq 403726925834 2026-02-13T02:09:19.588 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-13T02:09:19.588 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 403726925835 -lt 403726925834 2026-02-13T02:09:19.588 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:09:19.588 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:09:19.588 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:09:19.589 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:09:19.589 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:09:19.589 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:09:19.589 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:09:19.589 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:09:19.589 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:09:19.589 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:09:19.589 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:09:19.589 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:09:19.589 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:09:19.589 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:09:19.589 INFO:tasks.workunit.client.0.trial112.stderr:228218: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:09:19.590 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:09:19.590 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:09:19.590 INFO:tasks.workunit.client.0.trial112.stderr:228218: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:09:19.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-13T02:09:19.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-13T02:09:19.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 228217 2026-02-13T02:09:19.601 INFO:tasks.workunit.client.0.trial112.stderr:sd.9 seq 403726925835' 2026-02-13T02:09:19.601 INFO:tasks.workunit.client.0.trial112.stderr:228220: waiting osd.9 seq 403726925835 2026-02-13T02:09:19.602 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-13T02:09:19.602 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 403726925835 -lt 403726925835 2026-02-13T02:09:19.602 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:09:19.602 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:09:19.602 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:09:19.602 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:09:19.602 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:09:19.602 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:09:19.602 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:09:19.602 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:09:19.603 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:09:19.603 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:09:19.603 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:09:19.603 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:09:19.603 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:09:19.603 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:09:19.603 INFO:tasks.workunit.client.0.trial112.stderr:228220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:09:19.603 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:09:19.603 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:09:19.603 INFO:tasks.workunit.client.0.trial112.stderr:228220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:09:19.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-13T02:09:19.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-13T02:09:19.605 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-13T02:09:19.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-13T02:09:19.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:272: corrupt_and_repair_two: return_code=0 2026-02-13T02:09:19.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-13T02:09:19.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:275: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-13T02:09:19.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:276: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-13T02:09:19.624 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-13T02:09:19.624 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:09:19.625 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:09:19.625 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:09:19.625 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:09:19.625 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:09:19.626 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:09:19.626 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:09:19.626 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:09:19.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:09:19.940 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:09:19.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:09:19.943 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:09:19.944 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:09:19.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:09:19.945 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:09:19.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:09:19.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:09:19.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:09:19.947 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:09:19.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:09:19.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:09:19.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:09:19.987 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:09:19.987 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:09:19.987 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:09:19.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:09:19.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:09:19.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:09:19.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-13T02:09:19.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-13T02:09:19.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-13T02:09:19.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-13T02:09:19.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:09:19.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:09:19.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:09:19.991 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:09:19.991 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:09:19.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:09:19.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:09:19.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:09:19.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:09:19.993 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:09:19.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:09:19.994 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:09:19.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:09:19.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:09:19.996 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:09:19.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:09:19.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:09:19.998 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:09:19.998 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:09:19.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:09:20.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:09:20.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:09:20.001 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:09:20.002 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:09:20.002 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:09:20.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:09:20.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:09:20.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:09:20.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-13T02:09:20.004 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T02:09:20.004 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:09:20.005 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:09:20.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.35207 2026-02-13T02:09:20.006 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T02:09:20.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-13T02:09:20.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T02:09:20.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-13T02:09:20.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_replicated td/osd-scrub-repair 2026-02-13T02:09:20.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:94: TEST_corrupt_and_repair_replicated: local dir=td/osd-scrub-repair 2026-02-13T02:09:20.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:95: TEST_corrupt_and_repair_replicated: local poolname=rbd 2026-02-13T02:09:20.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:97: TEST_corrupt_and_repair_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2026-02-13T02:09:20.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-13T02:09:20.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-13T02:09:20.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-13T02:09:20.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-13T02:09:20.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-13T02:09:20.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2026-02-13T02:09:20.031 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-13T02:09:20.031 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:09:20.031 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:09:20.032 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:09:20.032 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:09:20.032 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:09:20.032 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:09:20.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2026-02-13T02:09:20.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-13T02:09:20.063 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-13T02:09:20.063 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:09:20.063 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:09:20.063 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-13T02:09:20.064 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-13T02:09:20.064 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:09:20.064 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:09:20.064 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:09:20.064 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:09:20.065 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:09:20.065 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:09:20.065 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:09:20.065 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:09:20.065 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get fsid 2026-02-13T02:09:20.123 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-13T02:09:20.124 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:09:20.124 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:09:20.124 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-13T02:09:20.124 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-13T02:09:20.125 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:09:20.125 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:09:20.125 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:09:20.125 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:09:20.126 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:09:20.126 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:09:20.126 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:09:20.126 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:09:20.126 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get mon_host 2026-02-13T02:09:20.183 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:98: TEST_corrupt_and_repair_replicated: run_mgr td/osd-scrub-repair x 2026-02-13T02:09:20.183 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-13T02:09:20.183 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-13T02:09:20.183 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-13T02:09:20.183 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-13T02:09:20.183 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-13T02:09:20.184 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-13T02:09:20.280 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-13T02:09:20.280 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:09:20.280 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:09:20.280 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:09:20.281 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:09:20.281 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:09:20.281 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:09:20.281 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:09:20.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:09:20.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:99: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 0 2026-02-13T02:09:20.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:09:20.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:09:20.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-13T02:09:20.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:09:20.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:09:20.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:09:20.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:09:20.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:09:20.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:09:20.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:09:20.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:09:20.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:09:20.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:09:20.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:09:20.304 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:09:20.304 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:09:20.304 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:09:20.304 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:09:20.304 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:09:20.304 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:09:20.304 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:09:20.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:09:20.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:09:20.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:09:20.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:09:20.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:09:20.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:09:20.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:09:20.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:09:20.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:09:20.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:09:20.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:09:20.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:09:20.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:09:20.307 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:09:20.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f46da909-e788-433c-970c-e6b82f280599 2026-02-13T02:09:20.307 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 f46da909-e788-433c-970c-e6b82f280599 2026-02-13T02:09:20.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 f46da909-e788-433c-970c-e6b82f280599' 2026-02-13T02:09:20.308 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:09:20.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDQh45pYqtjEhAAEivK1/b1CfccRUCxc4o5gw== 2026-02-13T02:09:20.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDQh45pYqtjEhAAEivK1/b1CfccRUCxc4o5gw=="}' 2026-02-13T02:09:20.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f46da909-e788-433c-970c-e6b82f280599 -i td/osd-scrub-repair/0/new.json 2026-02-13T02:09:20.409 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:09:20.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-13T02:09:20.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDQh45pYqtjEhAAEivK1/b1CfccRUCxc4o5gw== --osd-uuid f46da909-e788-433c-970c-e6b82f280599 2026-02-13T02:09:20.434 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:09:20.425+0000 7fa5690dba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:09:20.436 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:09:20.425+0000 7fa5690dba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:09:20.437 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:09:20.425+0000 7fa5690dba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:09:20.438 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:09:20.425+0000 7fa5690dba40 -1 bdev(0x55745386ec00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-13T02:09:20.438 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:09:20.425+0000 7fa5690dba40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-13T02:09:21.129 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-13T02:09:21.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:09:21.130 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T02:09:21.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-13T02:09:21.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:09:21.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-13T02:09:21.227 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T02:09:21.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:09:21.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:09:21.228 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:09:21.228 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:09:21.241 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:09:21.229+0000 7f317f7caa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:09:21.242 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:09:21.233+0000 7f317f7caa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:09:21.243 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:09:21.233+0000 7f317f7caa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:09:21.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-13T02:09:21.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:09:21.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:09:21.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:09:21.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:09:21.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:09:21.326 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:09:21.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:09:21.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:09:21.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:09:21.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:09:21.608 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:09:21.597+0000 7f317f7caa40 -1 Falling back to public interface 2026-02-13T02:09:22.311 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:09:22.301+0000 7f317f7caa40 -1 osd.0 0 log_to_monitors true 2026-02-13T02:09:22.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:09:22.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:09:22.419 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:09:22.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:09:22.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:09:22.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:09:22.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:09:23.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:09:23.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:09:23.551 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:09:23.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:09:23.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:09:23.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:09:23.711 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3495551283,v1:127.0.0.1:6801/3495551283] [v2:127.0.0.1:6802/3495551283,v1:127.0.0.1:6803/3495551283] exists,up f46da909-e788-433c-970c-e6b82f280599 2026-02-13T02:09:23.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:09:23.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:09:23.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:09:23.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:100: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 1 2026-02-13T02:09:23.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:09:23.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:09:23.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-13T02:09:23.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:09:23.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:09:23.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:09:23.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:09:23.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:09:23.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:09:23.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:09:23.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:09:23.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:09:23.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:09:23.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:09:23.714 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:09:23.714 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:09:23.714 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:09:23.715 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:09:23.715 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:09:23.715 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:09:23.715 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:09:23.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:09:23.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:09:23.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:09:23.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:09:23.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:09:23.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:09:23.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:09:23.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:09:23.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:09:23.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:09:23.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:09:23.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:09:23.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:09:23.716 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:09:23.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b567109a-6fc2-4113-bb21-7f8ca4d4d5b8 2026-02-13T02:09:23.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 b567109a-6fc2-4113-bb21-7f8ca4d4d5b8' 2026-02-13T02:09:23.716 INFO:tasks.workunit.client.0.trial112.stdout:add osd1 b567109a-6fc2-4113-bb21-7f8ca4d4d5b8 2026-02-13T02:09:23.717 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:09:23.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDTh45ptTjuKhAAj7y+AwaYMRc+D2ydrsdFkw== 2026-02-13T02:09:23.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDTh45ptTjuKhAAj7y+AwaYMRc+D2ydrsdFkw=="}' 2026-02-13T02:09:23.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b567109a-6fc2-4113-bb21-7f8ca4d4d5b8 -i td/osd-scrub-repair/1/new.json 2026-02-13T02:09:23.891 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:09:23.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-13T02:09:23.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDTh45ptTjuKhAAj7y+AwaYMRc+D2ydrsdFkw== --osd-uuid b567109a-6fc2-4113-bb21-7f8ca4d4d5b8 2026-02-13T02:09:23.927 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:09:23.917+0000 7f5e606bca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:09:23.929 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:09:23.917+0000 7f5e606bca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:09:23.930 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:09:23.921+0000 7f5e606bca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:09:23.931 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:09:23.921+0000 7f5e606bca40 -1 bdev(0x565134bfac00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-13T02:09:23.931 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:09:23.921+0000 7f5e606bca40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-13T02:09:24.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-13T02:09:24.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:09:24.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-13T02:09:24.567 INFO:tasks.workunit.client.0.trial112.stdout:adding osd1 key to auth repository 2026-02-13T02:09:24.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:09:24.779 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T02:09:24.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-13T02:09:24.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:09:24.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:09:24.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:09:24.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:09:24.803 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:09:24.793+0000 7f9c8d6a9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:09:24.805 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:09:24.793+0000 7f9c8d6a9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:09:24.806 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:09:24.797+0000 7f9c8d6a9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:09:24.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-13T02:09:24.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:09:24.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:09:24.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:09:24.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:09:24.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:09:24.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:09:24.940 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:09:24.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:09:24.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:09:25.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:09:25.232 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:09:25.221+0000 7f9c8d6a9a40 -1 Falling back to public interface 2026-02-13T02:09:25.832 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:09:25.821+0000 7f9c8d6a9a40 -1 osd.1 0 log_to_monitors true 2026-02-13T02:09:26.104 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:09:26.104 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:09:26.104 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:09:26.105 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:09:26.105 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:09:26.105 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:09:26.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:09:27.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:09:27.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:09:27.264 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:09:27.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:09:27.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:09:27.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:09:27.422 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/27358668,v1:127.0.0.1:6811/27358668] [v2:127.0.0.1:6812/27358668,v1:127.0.0.1:6813/27358668] exists,up b567109a-6fc2-4113-bb21-7f8ca4d4d5b8 2026-02-13T02:09:27.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:09:27.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:09:27.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:09:27.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:101: TEST_corrupt_and_repair_replicated: create_rbd_pool 2026-02-13T02:09:27.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-13T02:09:27.580 INFO:tasks.workunit.client.0.trial112.stderr:pool 'rbd' does not exist 2026-02-13T02:09:27.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-13T02:09:27.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-13T02:09:27.810 INFO:tasks.workunit.client.0.trial112.stderr:pool 'rbd' created 2026-02-13T02:09:27.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:09:28.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-13T02:09:29.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:102: TEST_corrupt_and_repair_replicated: wait_for_clean 2026-02-13T02:09:29.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:09:29.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:09:29.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:09:29.426 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:09:29.427 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:09:29.427 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:09:29.427 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:09:29.427 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:09:29.427 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:09:29.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:09:29.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:09:29.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:09:29.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:09:29.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:09:29.546 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:09:29.705 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:09:29.705 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:09:29.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:09:29.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:29.706 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:09:29.779 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-13T02:09:29.779 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-13T02:09:29.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-13T02:09:29.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:29.780 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:09:29.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672962 2026-02-13T02:09:29.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672962 2026-02-13T02:09:29.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-42949672962' 2026-02-13T02:09:29.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:29.852 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-13T02:09:29.852 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:29.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:09:29.854 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-13T02:09:29.854 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:29.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-13T02:09:29.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-13T02:09:29.856 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836483 2026-02-13T02:09:29.856 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:09:30.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-13T02:09:30.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:09:31.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:09:31.029 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:09:31.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-13T02:09:31.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:31.194 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672962 2026-02-13T02:09:31.195 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:31.196 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:09:31.197 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672962 2026-02-13T02:09:31.197 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:31.198 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672962 2026-02-13T02:09:31.199 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 42949672962 2026-02-13T02:09:31.199 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672962' 2026-02-13T02:09:31.199 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:09:31.359 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672962 -lt 42949672962 2026-02-13T02:09:31.359 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:09:31.360 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:09:31.360 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:09:31.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-13T02:09:31.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:09:31.555 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:09:31.555 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:09:31.555 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:09:31.555 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:09:31.555 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:09:31.555 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:09:31.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-13T02:09:31.711 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:09:31.711 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:09:31.711 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:09:31.910 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-13T02:09:31.910 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:09:31.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:09:31.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:104: TEST_corrupt_and_repair_replicated: add_something td/osd-scrub-repair rbd 2026-02-13T02:09:31.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:09:31.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2026-02-13T02:09:31.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2026-02-13T02:09:31.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:09:31.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:09:31.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:09:32.116 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:09:32.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:09:32.321 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:09:32.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:09:32.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:09:32.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-13T02:09:32.360 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:105: TEST_corrupt_and_repair_replicated: get_not_primary rbd SOMETHING 2026-02-13T02:09:32.360 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=rbd 2026-02-13T02:09:32.360 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-13T02:09:32.361 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary rbd SOMETHING 2026-02-13T02:09:32.361 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2026-02-13T02:09:32.361 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-13T02:09:32.361 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd SOMETHING 2026-02-13T02:09:32.361 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T02:09:32.519 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-13T02:09:32.520 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map rbd SOMETHING 2026-02-13T02:09:32.520 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-13T02:09:32.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:105: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2026-02-13T02:09:32.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:286: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2026-02-13T02:09:32.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: corrupt_and_repair_one: local poolname=rbd 2026-02-13T02:09:32.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:288: corrupt_and_repair_one: local osd=0 2026-02-13T02:09:32.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:293: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2026-02-13T02:09:32.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:09:32.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:09:32.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T02:09:32.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:09:32.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2026-02-13T02:09:32.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:09:32.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:09:32.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T02:09:32.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:09:32.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T02:09:32.687 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:09:32.687 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:09:32.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:09:32.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:09:32.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:09:32.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:09:32.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2026-02-13T02:09:32.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:09:32.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:09:32.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:09:32.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:09:32.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:09:32.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2026-02-13T02:09:33.189 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:eb822e21:::SOMETHING:head# 2026-02-13T02:09:33.271 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-13T02:09:33.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:09:33.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:09:33.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:09:33.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:09:33.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:09:33.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:09:33.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:09:33.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:09:33.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:09:33.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:09:33.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:09:33.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:09:33.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:09:33.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:09:33.272 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:09:33.272 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:09:33.273 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:09:33.273 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:09:33.273 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:09:33.273 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:09:33.274 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:09:33.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:09:33.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:09:33.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:09:33.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:09:33.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:09:33.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:09:33.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:09:33.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:09:33.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:09:33.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:09:33.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:09:33.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:09:33.274 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T02:09:33.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:09:33.275 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T02:09:33.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:09:33.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:09:33.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:09:33.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:09:33.292 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:09:33.281+0000 7fed64a5ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:09:33.293 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:09:33.281+0000 7fed64a5ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:09:33.294 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:09:33.285+0000 7fed64a5ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:09:33.435 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:09:33.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:09:33.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:09:33.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:09:33.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:09:33.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:09:33.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:09:33.436 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:09:33.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:09:33.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:09:33.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:09:33.727 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:09:33.717+0000 7fed64a5ba40 -1 Falling back to public interface 2026-02-13T02:09:34.337 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:09:34.325+0000 7fed64a5ba40 -1 osd.0 18 log_to_monitors true 2026-02-13T02:09:34.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:09:34.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:09:34.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:09:34.593 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:09:34.594 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:09:34.594 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:09:34.751 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 22 up_thru 22 down_at 19 last_clean_interval [5,18) [v2:127.0.0.1:6800/3215640939,v1:127.0.0.1:6801/3215640939] [v2:127.0.0.1:6802/3215640939,v1:127.0.0.1:6803/3215640939] exists,up f46da909-e788-433c-970c-e6b82f280599 2026-02-13T02:09:34.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:09:34.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:09:34.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:09:34.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:09:34.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:09:34.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:09:34.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:09:34.752 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:09:34.753 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:09:34.753 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:09:34.753 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:09:34.753 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:09:34.753 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:09:34.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:09:34.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:09:34.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:09:34.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:09:34.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:09:34.865 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:09:35.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:09:35.022 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:09:35.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:09:35.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:35.023 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:09:35.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280514 2026-02-13T02:09:35.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280514 2026-02-13T02:09:35.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280514' 2026-02-13T02:09:35.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:35.098 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:09:35.170 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672964 2026-02-13T02:09:35.170 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672964 2026-02-13T02:09:35.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280514 1-42949672964' 2026-02-13T02:09:35.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:35.171 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-94489280514 2026-02-13T02:09:35.171 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:35.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:09:35.173 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-94489280514 2026-02-13T02:09:35.173 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:35.175 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280514 2026-02-13T02:09:35.175 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 94489280514' 2026-02-13T02:09:35.175 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 94489280514 2026-02-13T02:09:35.176 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:09:35.339 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 94489280514 2026-02-13T02:09:35.339 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:09:36.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:09:36.341 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:09:36.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 94489280514 2026-02-13T02:09:36.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:09:37.506 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:09:37.506 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:09:37.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280514 -lt 94489280514 2026-02-13T02:09:37.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:37.673 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672964 2026-02-13T02:09:37.673 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:37.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:09:37.675 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672964 2026-02-13T02:09:37.675 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:37.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672964 2026-02-13T02:09:37.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2026-02-13T02:09:37.677 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 42949672964 2026-02-13T02:09:37.677 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:09:37.847 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672964 -lt 42949672964 2026-02-13T02:09:37.847 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:09:37.848 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:09:37.848 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:09:38.050 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-13T02:09:38.050 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:09:38.051 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:09:38.052 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:09:38.052 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:09:38.052 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:09:38.052 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:09:38.052 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:09:38.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-13T02:09:38.212 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:09:38.213 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:09:38.213 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:09:38.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-13T02:09:38.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:09:38.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:09:38.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: get_pg rbd SOMETHING 2026-02-13T02:09:38.408 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=rbd 2026-02-13T02:09:38.408 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-13T02:09:38.408 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map rbd SOMETHING 2026-02-13T02:09:38.408 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T02:09:38.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: local pg=1.3 2026-02-13T02:09:38.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: corrupt_and_repair_one: repair 1.3 2026-02-13T02:09:38.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=1.3 2026-02-13T02:09:38.567 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 1.3 2026-02-13T02:09:38.567 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-13T02:09:38.568 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:09:38.568 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:09:38.568 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-13T02:09:38.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-13T02:09:27.802011+0000 2026-02-13T02:09:38.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 1.3 2026-02-13T02:09:38.883 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 1.3 on osd.1 to repair 2026-02-13T02:09:38.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 1.3 2026-02-13T02:09:27.802011+0000 2026-02-13T02:09:38.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.3 2026-02-13T02:09:38.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:09:27.802011+0000 2026-02-13T02:09:38.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:09:38.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:09:38.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:09:38.895 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-13T02:09:38.895 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-13T02:09:38.895 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:09:38.895 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:09:38.895 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-13T02:09:39.054 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:09:27.802011+0000 '>' 2026-02-13T02:09:27.802011+0000 2026-02-13T02:09:39.055 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:09:40.055 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:09:40.055 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:09:40.056 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-13T02:09:40.056 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-13T02:09:40.056 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:09:40.056 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:09:40.057 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-13T02:09:40.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:09:27.802011+0000 '>' 2026-02-13T02:09:27.802011+0000 2026-02-13T02:09:40.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:09:41.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:09:41.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:09:41.217 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-13T02:09:41.217 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-13T02:09:41.217 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:09:41.217 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:09:41.217 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-13T02:09:41.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:09:27.802011+0000 '>' 2026-02-13T02:09:27.802011+0000 2026-02-13T02:09:41.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:09:42.376 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:09:42.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:09:42.378 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-13T02:09:42.378 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-13T02:09:42.378 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:09:42.378 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:09:42.378 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-13T02:09:42.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:09:27.802011+0000 '>' 2026-02-13T02:09:27.802011+0000 2026-02-13T02:09:42.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:09:43.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:09:43.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:09:43.536 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-13T02:09:43.536 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-13T02:09:43.536 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:09:43.537 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:09:43.537 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-13T02:09:43.694 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:09:39.870338+0000 '>' 2026-02-13T02:09:27.802011+0000 2026-02-13T02:09:43.694 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:09:43.694 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-13T02:09:43.694 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:09:43.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:09:43.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:09:43.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:09:43.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-13T02:09:43.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:09:43.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:09:43.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:09:43.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:09:43.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:09:43.695 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:09:43.695 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:09:43.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:09:43.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:09:43.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:09:44.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:09:44.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-13T02:09:44.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:09:44.004 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:09:44.004 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:09:44.004 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:09:44.004 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:09:44.004 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2026-02-13T02:09:44.077 INFO:tasks.workunit.client.0.trial112.stdout:_ 2026-02-13T02:09:44.078 INFO:tasks.workunit.client.0.trial112.stdout:snapset 2026-02-13T02:09:44.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-13T02:09:44.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:09:44.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:09:44.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:09:44.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:09:44.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:09:44.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:09:44.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:09:44.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:09:44.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:09:44.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:09:44.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:09:44.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:09:44.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:09:44.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:09:44.165 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:09:44.165 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:09:44.165 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:09:44.166 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:09:44.166 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:09:44.166 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:09:44.166 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:09:44.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:09:44.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:09:44.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:09:44.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:09:44.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:09:44.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:09:44.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:09:44.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:09:44.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:09:44.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:09:44.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:09:44.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:09:44.169 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:09:44.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:09:44.169 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:09:44.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:09:44.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:09:44.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:09:44.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:09:44.185 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:09:44.173+0000 7f4e09f34a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:09:44.187 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:09:44.177+0000 7f4e09f34a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:09:44.188 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:09:44.177+0000 7f4e09f34a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:09:44.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:09:44.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:09:44.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:09:44.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:09:44.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:09:44.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:09:44.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:09:44.327 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:09:44.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:09:44.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:09:44.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:09:44.516 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:09:44.505+0000 7f4e09f34a40 -1 Falling back to public interface 2026-02-13T02:09:45.116 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:09:45.105+0000 7f4e09f34a40 -1 osd.1 25 log_to_monitors true 2026-02-13T02:09:45.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:09:45.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:09:45.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:09:45.495 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:09:45.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:09:45.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:09:45.658 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 28 up_thru 28 down_at 25 last_clean_interval [10,25) [v2:127.0.0.1:6810/1206690707,v1:127.0.0.1:6811/1206690707] [v2:127.0.0.1:6812/1206690707,v1:127.0.0.1:6813/1206690707] exists,up b567109a-6fc2-4113-bb21-7f8ca4d4d5b8 2026-02-13T02:09:45.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:09:45.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:09:45.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:09:45.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:09:45.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:09:45.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:09:45.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:09:45.659 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:09:45.660 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:09:45.660 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:09:45.660 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:09:45.660 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:09:45.660 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:09:45.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:09:45.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:09:45.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:09:45.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:09:45.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:09:45.765 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:09:45.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:09:45.922 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:09:45.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:09:45.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:45.922 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:09:46.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280517 2026-02-13T02:09:46.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280517 2026-02-13T02:09:46.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280517' 2026-02-13T02:09:46.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:46.000 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:09:46.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=120259084290 2026-02-13T02:09:46.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 120259084290 2026-02-13T02:09:46.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280517 1-120259084290' 2026-02-13T02:09:46.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:46.077 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-94489280517 2026-02-13T02:09:46.077 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:46.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:09:46.079 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-94489280517 2026-02-13T02:09:46.079 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:46.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280517 2026-02-13T02:09:46.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 94489280517' 2026-02-13T02:09:46.081 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 94489280517 2026-02-13T02:09:46.081 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:09:46.243 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280516 -lt 94489280517 2026-02-13T02:09:46.243 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:09:47.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:09:47.245 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:09:47.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280517 -lt 94489280517 2026-02-13T02:09:47.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:47.414 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-120259084290 2026-02-13T02:09:47.414 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:47.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:09:47.416 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-120259084290 2026-02-13T02:09:47.416 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:47.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=120259084290 2026-02-13T02:09:47.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 120259084290' 2026-02-13T02:09:47.418 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 120259084290 2026-02-13T02:09:47.418 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:09:47.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 120259084290 -lt 120259084290 2026-02-13T02:09:47.591 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:09:47.592 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:09:47.592 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:09:47.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-13T02:09:47.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:09:47.789 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:09:47.789 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:09:47.789 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:09:47.789 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:09:47.790 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:09:47.790 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:09:47.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-13T02:09:47.946 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:09:47.946 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:09:47.946 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:09:48.152 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-13T02:09:48.152 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:09:48.152 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:09:48.152 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2026-02-13T02:09:48.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-13T02:09:48.173 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:107: TEST_corrupt_and_repair_replicated: get_primary rbd SOMETHING 2026-02-13T02:09:48.173 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2026-02-13T02:09:48.173 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-13T02:09:48.174 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T02:09:48.174 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd SOMETHING 2026-02-13T02:09:48.339 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:107: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 1 2026-02-13T02:09:48.339 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:286: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2026-02-13T02:09:48.339 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: corrupt_and_repair_one: local poolname=rbd 2026-02-13T02:09:48.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:288: corrupt_and_repair_one: local osd=1 2026-02-13T02:09:48.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:293: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2026-02-13T02:09:48.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:09:48.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:09:48.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:09:48.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:09:48.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2026-02-13T02:09:48.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:09:48.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:09:48.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:09:48.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:09:48.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:09:48.341 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:09:48.341 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:09:48.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:09:48.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:09:48.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:09:48.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:09:48.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2026-02-13T02:09:48.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:09:48.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:09:48.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:09:48.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:09:48.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:09:48.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2026-02-13T02:09:48.841 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:eb822e21:::SOMETHING:head# 2026-02-13T02:09:48.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-13T02:09:48.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:09:48.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:09:48.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:09:48.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:09:48.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:09:48.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:09:48.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:09:48.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:09:48.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:09:48.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:09:48.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:09:48.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:09:48.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:09:48.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:09:48.942 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:09:48.942 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:09:48.942 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:09:48.942 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:09:48.942 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:09:48.942 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:09:48.942 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:09:48.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:09:48.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:09:48.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:09:48.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:09:48.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:09:48.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:09:48.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:09:48.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:09:48.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:09:48.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:09:48.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:09:48.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:09:48.944 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:09:48.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:09:48.945 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:09:48.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:09:48.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:09:48.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:09:48.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:09:48.965 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:09:48.953+0000 7fc1acc4ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:09:48.966 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:09:48.957+0000 7fc1acc4ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:09:48.967 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:09:48.957+0000 7fc1acc4ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:09:49.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:09:49.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:09:49.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:09:49.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:09:49.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:09:49.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:09:49.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:09:49.110 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:09:49.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:09:49.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:09:49.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:09:49.344 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:09:49.333+0000 7fc1acc4ea40 -1 Falling back to public interface 2026-02-13T02:09:49.998 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:09:49.985+0000 7fc1acc4ea40 -1 osd.1 31 log_to_monitors true 2026-02-13T02:09:50.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:09:50.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:09:50.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:09:50.274 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:09:50.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:09:50.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:09:50.429 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 35 up_thru 35 down_at 32 last_clean_interval [28,31) [v2:127.0.0.1:6810/1314146161,v1:127.0.0.1:6811/1314146161] [v2:127.0.0.1:6812/1314146161,v1:127.0.0.1:6813/1314146161] exists,up b567109a-6fc2-4113-bb21-7f8ca4d4d5b8 2026-02-13T02:09:50.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:09:50.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:09:50.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:09:50.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:09:50.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:09:50.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:09:50.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:09:50.430 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:09:50.431 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:09:50.431 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:09:50.431 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:09:50.431 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:09:50.432 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:09:50.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:09:50.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:09:50.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:09:50.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:09:50.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:09:50.545 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:09:50.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:09:50.712 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:09:50.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:09:50.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:50.713 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:09:50.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280519 2026-02-13T02:09:50.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280519 2026-02-13T02:09:50.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280519' 2026-02-13T02:09:50.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:50.786 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:09:50.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855362 2026-02-13T02:09:50.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855362 2026-02-13T02:09:50.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280519 1-150323855362' 2026-02-13T02:09:50.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:50.859 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-94489280519 2026-02-13T02:09:50.859 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:50.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:09:50.861 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-94489280519 2026-02-13T02:09:50.862 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:50.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280519 2026-02-13T02:09:50.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 94489280519' 2026-02-13T02:09:50.863 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 94489280519 2026-02-13T02:09:50.863 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:09:51.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280518 -lt 94489280519 2026-02-13T02:09:51.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:09:52.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:09:52.028 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:09:52.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280518 -lt 94489280519 2026-02-13T02:09:52.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:09:53.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:09:53.193 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:09:53.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280519 -lt 94489280519 2026-02-13T02:09:53.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:53.356 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-150323855362 2026-02-13T02:09:53.356 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:53.357 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:09:53.358 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-150323855362 2026-02-13T02:09:53.358 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:53.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855362 2026-02-13T02:09:53.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 150323855362' 2026-02-13T02:09:53.360 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 150323855362 2026-02-13T02:09:53.360 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:09:53.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855362 -lt 150323855362 2026-02-13T02:09:53.530 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:09:53.531 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:09:53.531 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:09:53.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-13T02:09:53.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:09:53.725 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:09:53.725 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:09:53.725 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:09:53.725 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:09:53.725 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:09:53.725 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:09:53.886 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-13T02:09:53.887 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:09:53.887 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:09:53.888 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:09:54.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-13T02:09:54.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:09:54.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:09:54.091 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: get_pg rbd SOMETHING 2026-02-13T02:09:54.091 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=rbd 2026-02-13T02:09:54.091 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-13T02:09:54.091 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map rbd SOMETHING 2026-02-13T02:09:54.091 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T02:09:54.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: local pg=1.3 2026-02-13T02:09:54.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: corrupt_and_repair_one: repair 1.3 2026-02-13T02:09:54.258 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=1.3 2026-02-13T02:09:54.258 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 1.3 2026-02-13T02:09:54.259 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-13T02:09:54.259 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:09:54.259 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:09:54.259 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-13T02:09:54.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-13T02:09:39.870338+0000 2026-02-13T02:09:54.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 1.3 2026-02-13T02:09:54.567 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 1.3 on osd.1 to repair 2026-02-13T02:09:54.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 1.3 2026-02-13T02:09:39.870338+0000 2026-02-13T02:09:54.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.3 2026-02-13T02:09:54.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:09:39.870338+0000 2026-02-13T02:09:54.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:09:54.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:09:54.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:09:54.578 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-13T02:09:54.578 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-13T02:09:54.578 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:09:54.578 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:09:54.579 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-13T02:09:54.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:09:39.870338+0000 '>' 2026-02-13T02:09:39.870338+0000 2026-02-13T02:09:54.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:09:55.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:09:55.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:09:55.743 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-13T02:09:55.743 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-13T02:09:55.743 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:09:55.743 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:09:55.744 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-13T02:09:55.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:09:39.870338+0000 '>' 2026-02-13T02:09:39.870338+0000 2026-02-13T02:09:55.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:09:56.904 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:09:56.904 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:09:56.905 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-13T02:09:56.905 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-13T02:09:56.905 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:09:56.905 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:09:56.905 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-13T02:09:57.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:09:54.960880+0000 '>' 2026-02-13T02:09:39.870338+0000 2026-02-13T02:09:57.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:09:57.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-13T02:09:57.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:09:57.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:09:57.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:09:57.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:09:57.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-13T02:09:57.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:09:57.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:09:57.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:09:57.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:09:57.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:09:57.069 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:09:57.069 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:09:57.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:09:57.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:09:57.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:09:57.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:09:57.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-13T02:09:57.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:09:57.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:09:57.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:09:57.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:09:57.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:09:57.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2026-02-13T02:09:57.440 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::deep_scrub_1.3:head#, (61) No data available 2026-02-13T02:09:57.440 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2026-02-13T02:09:57.441 INFO:tasks.workunit.client.0.trial112.stdout:_ 2026-02-13T02:09:57.441 INFO:tasks.workunit.client.0.trial112.stdout:snapset 2026-02-13T02:09:57.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-13T02:09:57.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:09:57.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:09:57.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:09:57.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:09:57.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:09:57.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:09:57.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:09:57.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:09:57.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:09:57.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:09:57.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:09:57.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:09:57.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:09:57.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:09:57.518 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:09:57.518 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:09:57.518 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:09:57.519 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:09:57.519 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:09:57.519 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:09:57.519 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:09:57.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:09:57.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:09:57.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:09:57.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:09:57.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:09:57.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:09:57.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:09:57.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:09:57.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:09:57.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:09:57.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:09:57.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:09:57.521 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:09:57.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:09:57.522 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:09:57.523 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:09:57.523 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:09:57.523 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:09:57.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:09:57.541 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:09:57.529+0000 7f7248f2ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:09:57.543 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:09:57.533+0000 7f7248f2ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:09:57.544 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:09:57.533+0000 7f7248f2ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:09:57.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:09:57.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:09:57.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:09:57.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:09:57.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:09:57.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:09:57.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:09:57.687 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:09:57.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:09:57.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:09:57.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:09:57.896 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:09:57.885+0000 7f7248f2ea40 -1 Falling back to public interface 2026-02-13T02:09:58.500 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:09:58.489+0000 7f7248f2ea40 -1 osd.1 36 log_to_monitors true 2026-02-13T02:09:58.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:09:58.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:09:58.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:09:58.851 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:09:58.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:09:58.852 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:09:59.009 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 40 up_thru 40 down_at 37 last_clean_interval [35,36) [v2:127.0.0.1:6810/3459840003,v1:127.0.0.1:6811/3459840003] [v2:127.0.0.1:6812/3459840003,v1:127.0.0.1:6813/3459840003] exists,up b567109a-6fc2-4113-bb21-7f8ca4d4d5b8 2026-02-13T02:09:59.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:09:59.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:09:59.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:09:59.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:09:59.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:09:59.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:09:59.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:09:59.009 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:09:59.010 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:09:59.010 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:09:59.010 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:09:59.011 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:09:59.011 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:09:59.123 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:09:59.123 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:09:59.123 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:09:59.123 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:09:59.123 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:09:59.123 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:09:59.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:09:59.286 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:09:59.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:09:59.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:59.286 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:09:59.359 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280521 2026-02-13T02:09:59.359 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280521 2026-02-13T02:09:59.359 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280521' 2026-02-13T02:09:59.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:09:59.360 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:09:59.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691842 2026-02-13T02:09:59.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691842 2026-02-13T02:09:59.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280521 1-171798691842' 2026-02-13T02:09:59.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:09:59.434 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-94489280521 2026-02-13T02:09:59.434 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:09:59.435 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:09:59.436 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-94489280521 2026-02-13T02:09:59.436 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:09:59.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280521 2026-02-13T02:09:59.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 94489280521' 2026-02-13T02:09:59.438 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 94489280521 2026-02-13T02:09:59.438 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:09:59.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280520 -lt 94489280521 2026-02-13T02:09:59.615 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:10:00.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:10:00.616 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:10:00.774 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280522 -lt 94489280521 2026-02-13T02:10:00.774 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:10:00.774 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-171798691842 2026-02-13T02:10:00.774 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:10:00.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:10:00.776 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-171798691842 2026-02-13T02:10:00.777 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:10:00.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691842 2026-02-13T02:10:00.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 171798691842' 2026-02-13T02:10:00.778 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 171798691842 2026-02-13T02:10:00.779 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:10:00.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691842 -lt 171798691842 2026-02-13T02:10:00.948 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:10:00.949 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:10:00.949 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:10:01.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-13T02:10:01.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:10:01.150 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:10:01.150 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:10:01.150 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:10:01.150 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:10:01.150 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:10:01.150 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:10:01.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-13T02:10:01.311 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:10:01.311 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:10:01.312 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:10:01.512 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-13T02:10:01.512 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:10:01.512 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:10:01.513 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2026-02-13T02:10:01.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-13T02:10:01.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-13T02:10:01.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:10:01.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:10:01.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:10:01.533 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:10:01.533 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:10:01.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:10:01.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:10:01.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:10:01.643 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:10:01.643 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:10:01.644 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:10:01.645 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:10:01.646 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:10:01.646 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:10:01.646 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:10:01.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:10:01.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:10:01.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:10:01.648 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:10:01.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:10:01.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:10:01.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:10:01.661 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:10:01.661 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:10:01.661 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:10:01.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:10:01.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:10:01.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:10:01.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-13T02:10:01.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-13T02:10:01.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-13T02:10:01.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-13T02:10:01.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:10:01.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:10:01.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:10:01.664 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:10:01.664 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:10:01.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:10:01.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:10:01.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:10:01.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:10:01.667 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:10:01.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:10:01.668 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:10:01.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:10:01.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:10:01.670 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:10:01.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:10:01.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:10:01.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:10:01.672 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:10:01.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:10:01.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:10:01.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:10:01.676 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:10:01.676 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:10:01.676 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:10:01.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:10:01.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:10:01.678 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:10:01.678 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-13T02:10:01.679 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T02:10:01.679 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:10:01.679 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:10:01.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.35207 2026-02-13T02:10:01.681 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T02:10:01.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-13T02:10:01.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T02:10:01.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-13T02:10:01.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_erasure_appends td/osd-scrub-repair 2026-02-13T02:10:01.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5912: TEST_corrupt_scrub_erasure_appends: corrupt_scrub_erasure td/osd-scrub-repair false 2026-02-13T02:10:01.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3736: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2026-02-13T02:10:01.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3737: corrupt_scrub_erasure: local allow_overwrites=false 2026-02-13T02:10:01.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3738: corrupt_scrub_erasure: local poolname=ecpool 2026-02-13T02:10:01.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3739: corrupt_scrub_erasure: local total_objs=7 2026-02-13T02:10:01.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3741: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2026-02-13T02:10:01.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-13T02:10:01.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-13T02:10:01.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-13T02:10:01.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-13T02:10:01.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-13T02:10:01.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-13T02:10:01.703 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-13T02:10:01.704 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:10:01.704 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:10:01.704 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:10:01.704 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:10:01.704 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:10:01.704 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:10:01.704 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-13T02:10:01.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-13T02:10:01.734 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-13T02:10:01.734 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:10:01.735 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:10:01.735 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-13T02:10:01.736 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-13T02:10:01.736 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:10:01.736 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:10:01.736 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:10:01.736 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:10:01.736 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:10:01.736 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:10:01.736 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:10:01.737 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:10:01.737 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get fsid 2026-02-13T02:10:01.793 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-13T02:10:01.793 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:10:01.793 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:10:01.793 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-13T02:10:01.793 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-13T02:10:01.794 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:10:01.794 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:10:01.795 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:10:01.795 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:10:01.795 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:10:01.795 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:10:01.795 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:10:01.795 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:10:01.795 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get mon_host 2026-02-13T02:10:01.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3742: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2026-02-13T02:10:01.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-13T02:10:01.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-13T02:10:01.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-13T02:10:01.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-13T02:10:01.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-13T02:10:01.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-13T02:10:01.940 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-13T02:10:01.940 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:10:01.940 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:10:01.941 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:10:01.941 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:10:01.942 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:10:01.942 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:10:01.942 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:10:01.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:10:01.963 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3743: corrupt_scrub_erasure: seq 0 2 2026-02-13T02:10:01.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3743: corrupt_scrub_erasure: for id in $(seq 0 2) 2026-02-13T02:10:01.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3744: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 0 2026-02-13T02:10:01.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:10:01.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:10:01.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-13T02:10:01.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:10:01.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:10:01.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:10:01.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:10:01.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:10:01.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:10:01.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:10:01.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:10:01.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:10:01.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:10:01.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:10:01.968 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:10:01.968 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:10:01.968 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:10:01.968 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:10:01.968 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:10:01.968 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:10:01.968 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:10:01.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:10:01.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:10:01.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:10:01.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:10:01.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:10:01.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:10:01.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:10:01.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:10:01.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:10:01.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:10:01.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:10:01.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:10:01.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:10:01.970 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:10:01.970 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e45fb68f-2f91-4f83-a0b4-08802390b8b5 2026-02-13T02:10:01.970 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 e45fb68f-2f91-4f83-a0b4-08802390b8b5' 2026-02-13T02:10:01.971 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 e45fb68f-2f91-4f83-a0b4-08802390b8b5 2026-02-13T02:10:01.971 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:10:01.980 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD5h45ptgDqORAAmVNYnMdoAWDVLlCVD5IjwQ== 2026-02-13T02:10:01.980 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD5h45ptgDqORAAmVNYnMdoAWDVLlCVD5IjwQ=="}' 2026-02-13T02:10:01.980 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e45fb68f-2f91-4f83-a0b4-08802390b8b5 -i td/osd-scrub-repair/0/new.json 2026-02-13T02:10:02.073 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:10:02.082 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-13T02:10:02.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD5h45ptgDqORAAmVNYnMdoAWDVLlCVD5IjwQ== --osd-uuid e45fb68f-2f91-4f83-a0b4-08802390b8b5 2026-02-13T02:10:02.099 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:02.089+0000 7f4b0e99ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:10:02.101 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:02.089+0000 7f4b0e99ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:10:02.102 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:02.089+0000 7f4b0e99ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:10:02.102 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:02.089+0000 7f4b0e99ca40 -1 bdev(0x5652a7110c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-13T02:10:02.102 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:02.089+0000 7f4b0e99ca40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-13T02:10:02.893 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-13T02:10:02.893 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:10:02.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-13T02:10:02.895 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T02:10:02.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:10:02.990 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T02:10:02.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-13T02:10:02.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:10:02.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:10:02.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:10:02.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:10:03.016 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:03.005+0000 7fc4858a9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:10:03.018 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:03.005+0000 7fc4858a9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:10:03.019 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:03.009+0000 7fc4858a9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:10:03.082 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-13T02:10:03.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:10:03.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:10:03.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:10:03.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:10:03.083 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:10:03.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:10:03.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:10:03.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:10:03.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:10:03.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:10:03.528 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:03.517+0000 7fc4858a9a40 -1 Falling back to public interface 2026-02-13T02:10:04.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:10:04.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:10:04.175 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:10:04.175 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:10:04.176 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:10:04.176 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:10:04.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:10:04.514 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:04.501+0000 7fc4858a9a40 -1 osd.0 0 log_to_monitors true 2026-02-13T02:10:05.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:10:05.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:10:05.265 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:10:05.266 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:10:05.266 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:10:05.266 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:10:05.425 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1672430520,v1:127.0.0.1:6801/1672430520] [v2:127.0.0.1:6802/1672430520,v1:127.0.0.1:6803/1672430520] exists,up e45fb68f-2f91-4f83-a0b4-08802390b8b5 2026-02-13T02:10:05.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:10:05.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:10:05.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:10:05.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3743: corrupt_scrub_erasure: for id in $(seq 0 2) 2026-02-13T02:10:05.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3744: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 1 2026-02-13T02:10:05.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:10:05.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:10:05.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-13T02:10:05.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:10:05.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:10:05.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:10:05.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:10:05.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:10:05.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:10:05.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:10:05.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:10:05.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:10:05.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:10:05.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:10:05.432 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:10:05.432 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:10:05.432 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:10:05.432 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:10:05.432 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:10:05.432 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:10:05.432 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:10:05.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:10:05.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:10:05.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:10:05.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:10:05.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:10:05.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:10:05.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:10:05.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:10:05.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:10:05.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:10:05.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:10:05.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:10:05.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:10:05.434 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:10:05.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=396635b3-6e66-4635-9535-fe3bc01784cc 2026-02-13T02:10:05.434 INFO:tasks.workunit.client.0.trial112.stdout:add osd1 396635b3-6e66-4635-9535-fe3bc01784cc 2026-02-13T02:10:05.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 396635b3-6e66-4635-9535-fe3bc01784cc' 2026-02-13T02:10:05.435 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:10:05.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD9h45pjRr4GRAAbM0fSzKHhiV3b5AYEDgh4g== 2026-02-13T02:10:05.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD9h45pjRr4GRAAbM0fSzKHhiV3b5AYEDgh4g=="}' 2026-02-13T02:10:05.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 396635b3-6e66-4635-9535-fe3bc01784cc -i td/osd-scrub-repair/1/new.json 2026-02-13T02:10:05.603 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:10:05.613 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-13T02:10:05.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD9h45pjRr4GRAAbM0fSzKHhiV3b5AYEDgh4g== --osd-uuid 396635b3-6e66-4635-9535-fe3bc01784cc 2026-02-13T02:10:05.631 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:05.621+0000 7f42e36b7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:10:05.632 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:05.621+0000 7f42e36b7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:10:05.633 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:05.621+0000 7f42e36b7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:10:05.634 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:05.621+0000 7f42e36b7a40 -1 bdev(0x55d4b3dc0c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-13T02:10:05.634 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:05.621+0000 7f42e36b7a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-13T02:10:06.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-13T02:10:06.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:10:06.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-13T02:10:06.247 INFO:tasks.workunit.client.0.trial112.stdout:adding osd1 key to auth repository 2026-02-13T02:10:06.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:10:06.445 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T02:10:06.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-13T02:10:06.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:10:06.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:10:06.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:10:06.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:10:06.469 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:06.457+0000 7fc7b7e9ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:10:06.470 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:06.461+0000 7fc7b7e9ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:10:06.471 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:06.461+0000 7fc7b7e9ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:10:06.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-13T02:10:06.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:10:06.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:10:06.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:10:06.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:10:06.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:10:06.609 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:10:06.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:10:06.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:10:06.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:10:06.767 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:10:06.844 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:06.833+0000 7fc7b7e9ca40 -1 Falling back to public interface 2026-02-13T02:10:07.460 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:07.449+0000 7fc7b7e9ca40 -1 osd.1 0 log_to_monitors true 2026-02-13T02:10:07.769 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:10:07.769 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:10:07.769 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:10:07.770 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:10:07.770 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:10:07.770 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:10:07.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:10:08.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:10:08.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:10:08.933 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:10:08.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:10:08.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:10:08.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:10:09.089 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3021540671,v1:127.0.0.1:6811/3021540671] [v2:127.0.0.1:6812/3021540671,v1:127.0.0.1:6813/3021540671] exists,up 396635b3-6e66-4635-9535-fe3bc01784cc 2026-02-13T02:10:09.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:10:09.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:10:09.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:10:09.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3743: corrupt_scrub_erasure: for id in $(seq 0 2) 2026-02-13T02:10:09.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3744: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 2 2026-02-13T02:10:09.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:10:09.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:10:09.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-13T02:10:09.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:10:09.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-13T02:10:09.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:10:09.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:10:09.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:10:09.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:10:09.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-13T02:10:09.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-13T02:10:09.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:10:09.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:10:09.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:10:09.094 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:10:09.094 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:10:09.094 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:10:09.094 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:10:09.094 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:10:09.094 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:10:09.094 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:10:09.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:10:09.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:10:09.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:10:09.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:10:09.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:10:09.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:10:09.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:10:09.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:10:09.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:10:09.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:10:09.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:10:09.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:10:09.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-13T02:10:09.096 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:10:09.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=92cde444-63c3-4cc0-ab5f-8698b24011e5 2026-02-13T02:10:09.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 92cde444-63c3-4cc0-ab5f-8698b24011e5' 2026-02-13T02:10:09.097 INFO:tasks.workunit.client.0.trial112.stdout:add osd2 92cde444-63c3-4cc0-ab5f-8698b24011e5 2026-02-13T02:10:09.097 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:10:09.107 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQABiI5pGSHXBRAATuCnA1o4n/SStWBj8Nb3/w== 2026-02-13T02:10:09.107 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQABiI5pGSHXBRAATuCnA1o4n/SStWBj8Nb3/w=="}' 2026-02-13T02:10:09.107 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 92cde444-63c3-4cc0-ab5f-8698b24011e5 -i td/osd-scrub-repair/2/new.json 2026-02-13T02:10:09.263 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:10:09.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-13T02:10:09.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQABiI5pGSHXBRAATuCnA1o4n/SStWBj8Nb3/w== --osd-uuid 92cde444-63c3-4cc0-ab5f-8698b24011e5 2026-02-13T02:10:09.292 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:09.281+0000 7fd83705ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:10:09.293 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:09.281+0000 7fd83705ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:10:09.294 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:09.281+0000 7fd83705ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:10:09.295 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:09.285+0000 7fd83705ba40 -1 bdev(0x55c22c2eec00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-13T02:10:09.295 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:09.285+0000 7fd83705ba40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-13T02:10:09.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-13T02:10:09.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:10:09.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-13T02:10:09.975 INFO:tasks.workunit.client.0.trial112.stdout:adding osd2 key to auth repository 2026-02-13T02:10:09.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:10:10.171 INFO:tasks.workunit.client.0.trial112.stdout:start osd.2 2026-02-13T02:10:10.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-13T02:10:10.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:10:10.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:10:10.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:10:10.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:10:10.195 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:10.184+0000 7fbeabff2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:10:10.197 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:10.184+0000 7fbeabff2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:10:10.198 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:10.184+0000 7fbeabff2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:10:10.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-13T02:10:10.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:10:10.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T02:10:10.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:10:10.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:10:10.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:10:10.338 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:10:10.339 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:10:10.339 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:10:10.339 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:10:10.497 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:10:10.612 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:10.600+0000 7fbeabff2a40 -1 Falling back to public interface 2026-02-13T02:10:11.245 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:11.232+0000 7fbeabff2a40 -1 osd.2 0 log_to_monitors true 2026-02-13T02:10:11.498 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:10:11.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:10:11.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:10:11.499 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:10:11.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:10:11.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:10:11.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:10:12.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:10:12.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:10:12.661 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:10:12.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:10:12.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:10:12.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:10:12.818 INFO:tasks.workunit.client.0.trial112.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3388502324,v1:127.0.0.1:6819/3388502324] [v2:127.0.0.1:6820/3388502324,v1:127.0.0.1:6821/3388502324] exists,up 92cde444-63c3-4cc0-ab5f-8698b24011e5 2026-02-13T02:10:12.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:10:12.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:10:12.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:10:12.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3746: corrupt_scrub_erasure: create_rbd_pool 2026-02-13T02:10:12.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-13T02:10:12.968 INFO:tasks.workunit.client.0.trial112.stderr:pool 'rbd' does not exist 2026-02-13T02:10:12.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-13T02:10:12.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-13T02:10:13.198 INFO:tasks.workunit.client.0.trial112.stderr:pool 'rbd' created 2026-02-13T02:10:13.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:10:14.210 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-13T02:10:14.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3747: corrupt_scrub_erasure: create_pool foo 1 2026-02-13T02:10:14.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 1 2026-02-13T02:10:14.689 INFO:tasks.workunit.client.0.trial112.stderr:pool 'foo' created 2026-02-13T02:10:14.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:10:15.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3749: corrupt_scrub_erasure: create_ec_pool ecpool false k=2 m=1 stripe_unit=2K --force 2026-02-13T02:10:15.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2522: create_ec_pool: local pool_name=ecpool 2026-02-13T02:10:15.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2523: create_ec_pool: shift 2026-02-13T02:10:15.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2524: create_ec_pool: local allow_overwrites=false 2026-02-13T02:10:15.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2525: create_ec_pool: shift 2026-02-13T02:10:15.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2527: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2026-02-13T02:10:15.912 INFO:tasks.workunit.client.0.trial112.stderr:load: isa 2026-02-13T02:10:15.928 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2529: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2026-02-13T02:10:15.928 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2026-02-13T02:10:16.218 INFO:tasks.workunit.client.0.trial112.stderr:pool 'ecpool' created 2026-02-13T02:10:16.232 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:10:17.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2531: create_ec_pool: '[' false = true ']' 2026-02-13T02:10:17.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2535: create_ec_pool: wait_for_clean 2026-02-13T02:10:17.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:10:17.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:10:17.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:10:17.235 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:10:17.235 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:10:17.235 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:10:17.236 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:10:17.236 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:10:17.236 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:10:17.348 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:10:17.348 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:10:17.348 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:10:17.348 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:10:17.348 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:10:17.348 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:10:17.510 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:10:17.510 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:10:17.510 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:10:17.510 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:10:17.511 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:10:17.511 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:10:17.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836484 2026-02-13T02:10:17.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836484 2026-02-13T02:10:17.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484' 2026-02-13T02:10:17.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:10:17.583 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:10:17.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705667 2026-02-13T02:10:17.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705667 2026-02-13T02:10:17.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-38654705667' 2026-02-13T02:10:17.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:10:17.659 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:10:17.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542147 2026-02-13T02:10:17.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542147 2026-02-13T02:10:17.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-38654705667 2-60129542147' 2026-02-13T02:10:17.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:10:17.734 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836484 2026-02-13T02:10:17.734 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:10:17.735 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:10:17.736 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836484 2026-02-13T02:10:17.736 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:10:17.737 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836484 2026-02-13T02:10:17.738 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836484 2026-02-13T02:10:17.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-13T02:10:17.738 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:10:17.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836484 2026-02-13T02:10:17.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:10:18.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:10:18.904 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:10:19.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836484 2026-02-13T02:10:19.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:10:19.064 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705667 2026-02-13T02:10:19.064 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:10:19.065 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:10:19.066 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705667 2026-02-13T02:10:19.066 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:10:19.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705667 2026-02-13T02:10:19.068 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705667 2026-02-13T02:10:19.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705667' 2026-02-13T02:10:19.068 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:10:19.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705667 -lt 38654705667 2026-02-13T02:10:19.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:10:19.240 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542147 2026-02-13T02:10:19.241 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:10:19.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:10:19.242 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542147 2026-02-13T02:10:19.242 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:10:19.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542147 2026-02-13T02:10:19.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542147' 2026-02-13T02:10:19.244 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542147 2026-02-13T02:10:19.244 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:10:19.402 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542147 -lt 60129542147 2026-02-13T02:10:19.402 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:10:19.403 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:10:19.403 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:10:19.594 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:10:19.594 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:10:19.595 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:10:19.595 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:10:19.595 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:10:19.595 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:10:19.595 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:10:19.595 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:10:19.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:10:19.751 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:10:19.752 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:10:19.752 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:10:19.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:10:19.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:10:19.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:10:19.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2536: create_ec_pool: return 0 2026-02-13T02:10:19.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3750: corrupt_scrub_erasure: wait_for_clean 2026-02-13T02:10:19.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:10:19.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:10:19.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:10:19.948 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:10:19.948 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:10:19.949 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:10:19.949 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:10:19.949 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:10:19.949 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:10:20.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:10:20.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:10:20.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:10:20.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:10:20.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:10:20.060 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:10:20.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:10:20.226 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:10:20.226 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:10:20.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:10:20.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:10:20.226 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:10:20.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836485 2026-02-13T02:10:20.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836485 2026-02-13T02:10:20.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485' 2026-02-13T02:10:20.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:10:20.299 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:10:20.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705669 2026-02-13T02:10:20.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705669 2026-02-13T02:10:20.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-38654705669' 2026-02-13T02:10:20.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:10:20.371 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:10:20.442 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542148 2026-02-13T02:10:20.442 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542148 2026-02-13T02:10:20.442 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-38654705669 2-60129542148' 2026-02-13T02:10:20.442 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:10:20.442 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836485 2026-02-13T02:10:20.443 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:10:20.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:10:20.444 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836485 2026-02-13T02:10:20.444 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:10:20.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836485 2026-02-13T02:10:20.446 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836485 2026-02-13T02:10:20.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-13T02:10:20.447 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:10:20.610 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836485 2026-02-13T02:10:20.610 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:10:21.611 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:10:21.612 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:10:21.774 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836485 2026-02-13T02:10:21.774 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:10:22.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:10:22.776 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:10:22.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836485 2026-02-13T02:10:22.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:10:22.942 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705669 2026-02-13T02:10:22.942 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:10:22.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:10:22.944 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705669 2026-02-13T02:10:22.945 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:10:22.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705669 2026-02-13T02:10:22.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705669' 2026-02-13T02:10:22.947 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705669 2026-02-13T02:10:22.947 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:10:23.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705669 -lt 38654705669 2026-02-13T02:10:23.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:10:23.118 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542148 2026-02-13T02:10:23.118 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:10:23.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:10:23.120 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542148 2026-02-13T02:10:23.120 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:10:23.121 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542148 2026-02-13T02:10:23.122 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542148 2026-02-13T02:10:23.123 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542148' 2026-02-13T02:10:23.123 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:10:23.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542148 -lt 60129542148 2026-02-13T02:10:23.299 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:10:23.299 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:10:23.299 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:10:23.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:10:23.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:10:23.497 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:10:23.498 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:10:23.498 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:10:23.498 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:10:23.498 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:10:23.498 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:10:23.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:10:23.661 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:10:23.661 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:10:23.661 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:10:23.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:10:23.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:10:23.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:10:23.858 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: seq 1 7 2026-02-13T02:10:23.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-13T02:10:23.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ1 2026-02-13T02:10:23.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2026-02-13T02:10:23.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:10:23.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-13T02:10:23.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ1 2026-02-13T02:10:23.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:10:23.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:10:23.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:10:24.099 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:10:24.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:10:24.300 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:10:24.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:10:24.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:10:24.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:10:24.337 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 1 % 2 2026-02-13T02:10:24.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=1 2026-02-13T02:10:24.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-13T02:10:24.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3761: corrupt_scrub_erasure: local payload=UVWXYZZZ 2026-02-13T02:10:24.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3762: corrupt_scrub_erasure: echo UVWXYZZZ 2026-02-13T02:10:24.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3763: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-13T02:10:24.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:10:24.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:10:24.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:10:24.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:10:24.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-13T02:10:24.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:10:24.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:10:24.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:10:24.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:10:24.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:10:24.341 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:10:24.341 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:10:24.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:10:24.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:10:24.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:10:24.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:10:24.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-13T02:10:24.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:10:24.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:10:24.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:10:24.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:10:24.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:10:24.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-13T02:10:24.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-13T02:10:24.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:10:24.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:10:24.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:10:24.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:10:24.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:10:24.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:10:24.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:10:24.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:10:24.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:10:24.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:10:24.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:10:24.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:10:24.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:10:24.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:10:24.934 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:10:24.934 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:10:24.934 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:10:24.934 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:10:24.934 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:10:24.934 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:10:24.934 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:10:24.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:10:24.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:10:24.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:10:24.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:10:24.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:10:24.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:10:24.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:10:24.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:10:24.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:10:24.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:10:24.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:10:24.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:10:24.937 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:10:24.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:10:24.937 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:10:24.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:10:24.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:10:24.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:10:24.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:10:24.957 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:24.944+0000 7fa2c287ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:10:24.958 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:24.944+0000 7fa2c287ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:10:24.959 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:24.948+0000 7fa2c287ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:10:25.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:10:25.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:10:25.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:10:25.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:10:25.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:10:25.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:10:25.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:10:25.098 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:10:25.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:10:25.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:10:25.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:10:25.292 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:25.280+0000 7fa2c287ba40 -1 Falling back to public interface 2026-02-13T02:10:25.927 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:25.916+0000 7fa2c287ba40 -1 osd.1 32 log_to_monitors true 2026-02-13T02:10:26.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:10:26.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:10:26.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:10:26.264 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:10:26.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:10:26.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:10:26.421 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 36 up_thru 36 down_at 33 last_clean_interval [9,32) [v2:127.0.0.1:6810/632098867,v1:127.0.0.1:6811/632098867] [v2:127.0.0.1:6812/632098867,v1:127.0.0.1:6813/632098867] exists,up 396635b3-6e66-4635-9535-fe3bc01784cc 2026-02-13T02:10:26.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:10:26.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:10:26.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:10:26.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:10:26.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:10:26.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:10:26.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:10:26.423 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:10:26.423 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:10:26.423 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:10:26.424 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:10:26.424 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:10:26.424 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:10:26.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:10:26.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:10:26.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:10:26.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:10:26.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:10:26.533 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:10:26.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:10:26.698 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:10:26.698 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:10:26.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:10:26.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:10:26.699 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:10:26.770 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836488 2026-02-13T02:10:26.770 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836488 2026-02-13T02:10:26.771 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488' 2026-02-13T02:10:26.771 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:10:26.771 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:10:26.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=154618822658 2026-02-13T02:10:26.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 154618822658 2026-02-13T02:10:26.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-154618822658' 2026-02-13T02:10:26.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:10:26.843 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:10:26.914 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542150 2026-02-13T02:10:26.914 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542150 2026-02-13T02:10:26.914 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-154618822658 2-60129542150' 2026-02-13T02:10:26.914 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:10:26.915 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836488 2026-02-13T02:10:26.915 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:10:26.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:10:26.916 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836488 2026-02-13T02:10:26.917 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:10:26.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836488 2026-02-13T02:10:26.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2026-02-13T02:10:26.918 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 21474836488 2026-02-13T02:10:26.919 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:10:27.079 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836488 2026-02-13T02:10:27.079 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:10:28.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:10:28.081 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:10:28.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836488 2026-02-13T02:10:28.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:10:29.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:10:29.248 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:10:29.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836488 2026-02-13T02:10:29.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:10:29.410 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-154618822658 2026-02-13T02:10:29.410 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:10:29.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:10:29.412 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-154618822658 2026-02-13T02:10:29.412 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:10:29.414 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=154618822658 2026-02-13T02:10:29.414 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 154618822658' 2026-02-13T02:10:29.414 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 154618822658 2026-02-13T02:10:29.414 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:10:29.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 154618822658 -lt 154618822658 2026-02-13T02:10:29.584 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:10:29.584 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542150 2026-02-13T02:10:29.584 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:10:29.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:10:29.587 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542150 2026-02-13T02:10:29.587 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:10:29.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542150 2026-02-13T02:10:29.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542150' 2026-02-13T02:10:29.589 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 60129542150 2026-02-13T02:10:29.589 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:10:29.760 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542151 -lt 60129542150 2026-02-13T02:10:29.761 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:10:29.762 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:10:29.762 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:10:29.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:10:29.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:10:29.965 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:10:29.965 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:10:29.965 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:10:29.965 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:10:29.965 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:10:29.966 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:10:30.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:10:30.124 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:10:30.125 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:10:30.125 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:10:30.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:10:30.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:10:30.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:10:30.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-13T02:10:30.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ2 2026-02-13T02:10:30.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2026-02-13T02:10:30.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:10:30.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-13T02:10:30.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ2 2026-02-13T02:10:30.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:10:30.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:10:30.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:10:30.476 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:10:30.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:10:30.678 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:10:30.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:10:30.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:10:30.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:10:30.719 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 2 % 2 2026-02-13T02:10:30.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=0 2026-02-13T02:10:30.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-13T02:10:30.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3768: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2026-02-13T02:10:30.723 INFO:tasks.workunit.client.0.trial112.stderr:1+0 records in 2026-02-13T02:10:30.724 INFO:tasks.workunit.client.0.trial112.stderr:1+0 records out 2026-02-13T02:10:30.724 INFO:tasks.workunit.client.0.trial112.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 8.7973e-05 s, 23.3 MB/s 2026-02-13T02:10:30.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3769: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-13T02:10:30.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:10:30.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:10:30.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T02:10:30.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:10:30.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-13T02:10:30.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:10:30.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:10:30.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T02:10:30.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:10:30.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T02:10:30.725 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:10:30.725 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:10:30.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:10:30.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:10:30.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:10:31.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:10:31.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-13T02:10:31.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:10:31.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:10:31.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:10:31.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:10:31.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:10:31.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-13T02:10:31.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-13T02:10:31.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:10:31.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:10:31.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:10:31.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:10:31.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:10:31.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:10:31.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:10:31.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:10:31.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:10:31.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:10:31.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:10:31.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:10:31.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:10:31.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:10:31.318 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:10:31.318 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:10:31.318 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:10:31.318 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:10:31.318 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:10:31.318 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:10:31.318 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:10:31.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:10:31.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:10:31.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:10:31.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:10:31.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:10:31.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:10:31.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:10:31.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:10:31.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:10:31.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:10:31.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:10:31.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:10:31.321 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T02:10:31.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:10:31.322 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T02:10:31.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:10:31.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:10:31.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:10:31.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:10:31.338 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:31.324+0000 7f1b4fbaea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:10:31.339 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:31.328+0000 7f1b4fbaea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:10:31.340 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:31.328+0000 7f1b4fbaea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:10:31.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:10:31.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:10:31.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:10:31.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:10:31.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:10:31.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:10:31.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:10:31.483 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:10:31.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:10:31.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:10:31.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:10:31.740 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:31.728+0000 7f1b4fbaea40 -1 Falling back to public interface 2026-02-13T02:10:32.330 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:32.316+0000 7f1b4fbaea40 -1 osd.0 39 log_to_monitors true 2026-02-13T02:10:32.642 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:10:32.643 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:10:32.643 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:10:32.643 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:10:32.643 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:10:32.643 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:10:32.798 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 43 up_thru 43 down_at 40 last_clean_interval [5,39) [v2:127.0.0.1:6800/3192759602,v1:127.0.0.1:6801/3192759602] [v2:127.0.0.1:6802/3192759602,v1:127.0.0.1:6803/3192759602] exists,up e45fb68f-2f91-4f83-a0b4-08802390b8b5 2026-02-13T02:10:32.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:10:32.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:10:32.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:10:32.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:10:32.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:10:32.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:10:32.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:10:32.799 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:10:32.799 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:10:32.800 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:10:32.800 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:10:32.800 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:10:32.800 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:10:32.908 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:10:32.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:10:32.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:10:32.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:10:32.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:10:32.909 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:10:33.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:10:33.071 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:10:33.071 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:10:33.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:10:33.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:10:33.071 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:10:33.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=184683593730 2026-02-13T02:10:33.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 184683593730 2026-02-13T02:10:33.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-184683593730' 2026-02-13T02:10:33.144 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:10:33.144 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:10:33.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=154618822660 2026-02-13T02:10:33.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 154618822660 2026-02-13T02:10:33.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-184683593730 1-154618822660' 2026-02-13T02:10:33.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:10:33.216 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:10:33.289 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542153 2026-02-13T02:10:33.290 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542153 2026-02-13T02:10:33.290 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-184683593730 1-154618822660 2-60129542153' 2026-02-13T02:10:33.290 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:10:33.290 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-184683593730 2026-02-13T02:10:33.291 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:10:33.291 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:10:33.292 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-184683593730 2026-02-13T02:10:33.292 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:10:33.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=184683593730 2026-02-13T02:10:33.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 184683593730' 2026-02-13T02:10:33.294 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 184683593730 2026-02-13T02:10:33.295 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:10:33.463 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 184683593730 2026-02-13T02:10:33.463 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:10:34.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:10:34.465 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:10:34.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 184683593730 -lt 184683593730 2026-02-13T02:10:34.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:10:34.629 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-154618822660 2026-02-13T02:10:34.630 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:10:34.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:10:34.632 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-154618822660 2026-02-13T02:10:34.632 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:10:34.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=154618822660 2026-02-13T02:10:34.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 154618822660' 2026-02-13T02:10:34.634 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 154618822660 2026-02-13T02:10:34.634 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:10:34.801 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 154618822660 -lt 154618822660 2026-02-13T02:10:34.801 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:10:34.802 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542153 2026-02-13T02:10:34.802 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:10:34.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:10:34.804 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542153 2026-02-13T02:10:34.804 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:10:34.806 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542153 2026-02-13T02:10:34.806 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542153' 2026-02-13T02:10:34.806 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 60129542153 2026-02-13T02:10:34.806 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:10:34.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542153 -lt 60129542153 2026-02-13T02:10:34.967 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:10:34.968 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:10:34.968 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:10:35.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:10:35.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:10:35.167 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:10:35.167 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:10:35.167 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:10:35.167 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:10:35.168 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:10:35.168 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:10:35.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:10:35.329 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:10:35.330 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:10:35.330 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:10:35.527 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:10:35.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:10:35.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:10:35.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-13T02:10:35.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ3 2026-02-13T02:10:35.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2026-02-13T02:10:35.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:10:35.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-13T02:10:35.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ3 2026-02-13T02:10:35.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:10:35.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:10:35.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:10:35.734 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:10:35.748 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:10:35.939 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:10:35.950 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:10:35.950 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:10:35.950 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:10:35.971 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 3 % 2 2026-02-13T02:10:35.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=1 2026-02-13T02:10:35.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-13T02:10:35.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3774: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2026-02-13T02:10:35.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:10:35.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:10:35.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:10:35.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:10:35.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2026-02-13T02:10:35.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:10:35.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:10:35.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:10:35.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:10:35.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:10:35.974 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:10:35.974 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:10:35.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:10:35.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:10:35.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:10:36.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:10:36.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2026-02-13T02:10:36.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:10:36.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:10:36.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:10:36.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:10:36.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:10:36.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2026-02-13T02:10:36.455 INFO:tasks.workunit.client.0.trial112.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2026-02-13T02:10:36.548 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-13T02:10:36.548 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:10:36.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:10:36.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:10:36.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:10:36.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:10:36.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:10:36.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:10:36.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:10:36.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:10:36.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:10:36.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:10:36.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:10:36.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:10:36.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:10:36.550 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:10:36.550 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:10:36.550 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:10:36.550 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:10:36.550 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:10:36.550 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:10:36.552 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:10:36.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:10:36.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:10:36.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:10:36.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:10:36.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:10:36.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:10:36.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:10:36.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:10:36.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:10:36.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:10:36.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:10:36.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:10:36.553 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:10:36.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:10:36.553 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:10:36.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:10:36.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:10:36.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:10:36.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:10:36.572 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:36.560+0000 7fd855ffea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:10:36.574 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:36.560+0000 7fd855ffea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:10:36.575 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:36.564+0000 7fd855ffea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:10:36.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:10:36.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:10:36.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:10:36.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:10:36.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:10:36.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:10:36.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:10:36.709 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:10:36.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:10:36.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:10:36.871 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:10:36.960 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:36.948+0000 7fd855ffea40 -1 Falling back to public interface 2026-02-13T02:10:37.514 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:37.500+0000 7fd855ffea40 -1 osd.1 47 log_to_monitors true 2026-02-13T02:10:37.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:10:37.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:10:37.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:10:37.874 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:10:37.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:10:37.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:10:38.035 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 51 up_thru 51 down_at 48 last_clean_interval [36,47) [v2:127.0.0.1:6810/3331197691,v1:127.0.0.1:6811/3331197691] [v2:127.0.0.1:6812/3331197691,v1:127.0.0.1:6813/3331197691] exists,up 396635b3-6e66-4635-9535-fe3bc01784cc 2026-02-13T02:10:38.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:10:38.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:10:38.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:10:38.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:10:38.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:10:38.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:10:38.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:10:38.036 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:10:38.037 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:10:38.037 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:10:38.037 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:10:38.037 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:10:38.037 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:10:38.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:10:38.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:10:38.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:10:38.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:10:38.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:10:38.146 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:10:38.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:10:38.310 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:10:38.310 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:10:38.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:10:38.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:10:38.311 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:10:38.383 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=184683593732 2026-02-13T02:10:38.383 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 184683593732 2026-02-13T02:10:38.383 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-184683593732' 2026-02-13T02:10:38.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:10:38.384 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:10:38.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=219043332098 2026-02-13T02:10:38.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 219043332098 2026-02-13T02:10:38.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-184683593732 1-219043332098' 2026-02-13T02:10:38.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:10:38.458 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:10:38.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542155 2026-02-13T02:10:38.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542155 2026-02-13T02:10:38.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-184683593732 1-219043332098 2-60129542155' 2026-02-13T02:10:38.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:10:38.530 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-184683593732 2026-02-13T02:10:38.530 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:10:38.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:10:38.532 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-184683593732 2026-02-13T02:10:38.532 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:10:38.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=184683593732 2026-02-13T02:10:38.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 184683593732' 2026-02-13T02:10:38.534 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 184683593732 2026-02-13T02:10:38.534 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:10:38.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 184683593731 -lt 184683593732 2026-02-13T02:10:38.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:10:39.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:10:39.699 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:10:39.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 184683593731 -lt 184683593732 2026-02-13T02:10:39.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:10:40.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:10:40.865 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:10:41.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 184683593732 -lt 184683593732 2026-02-13T02:10:41.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:10:41.032 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-219043332098 2026-02-13T02:10:41.032 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:10:41.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:10:41.035 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-219043332098 2026-02-13T02:10:41.035 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:10:41.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=219043332098 2026-02-13T02:10:41.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 219043332098' 2026-02-13T02:10:41.036 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 219043332098 2026-02-13T02:10:41.037 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:10:41.203 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 219043332098 -lt 219043332098 2026-02-13T02:10:41.203 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:10:41.203 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542155 2026-02-13T02:10:41.204 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:10:41.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:10:41.206 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542155 2026-02-13T02:10:41.206 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:10:41.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542155 2026-02-13T02:10:41.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542155' 2026-02-13T02:10:41.208 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 60129542155 2026-02-13T02:10:41.208 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:10:41.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542155 -lt 60129542155 2026-02-13T02:10:41.373 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:10:41.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:10:41.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:10:41.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:10:41.572 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:10:41.573 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:10:41.573 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:10:41.573 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:10:41.573 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:10:41.573 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:10:41.573 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:10:41.735 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:10:41.736 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:10:41.736 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:10:41.736 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:10:41.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:10:41.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:10:41.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:10:41.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-13T02:10:41.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ4 2026-02-13T02:10:41.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2026-02-13T02:10:41.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:10:41.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-13T02:10:41.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ4 2026-02-13T02:10:41.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:10:41.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:10:41.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:10:42.141 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:10:42.155 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:10:42.348 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:10:42.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:10:42.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:10:42.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:10:42.382 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 4 % 2 2026-02-13T02:10:42.383 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=0 2026-02-13T02:10:42.383 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-13T02:10:42.383 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3778: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2026-02-13T02:10:42.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3779: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2026-02-13T02:10:42.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3782: corrupt_scrub_erasure: echo -n bad-val 2026-02-13T02:10:42.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3783: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2026-02-13T02:10:42.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:10:42.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:10:42.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T02:10:42.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:10:42.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2026-02-13T02:10:42.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:10:42.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:10:42.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T02:10:42.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:10:42.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T02:10:42.431 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:10:42.431 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:10:42.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:10:42.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:10:42.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:10:42.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:10:42.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2026-02-13T02:10:42.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:10:42.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:10:42.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:10:42.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:10:42.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:10:42.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2026-02-13T02:10:43.004 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-13T02:10:43.004 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:10:43.004 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:10:43.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:10:43.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:10:43.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:10:43.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:10:43.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:10:43.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:10:43.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:10:43.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:10:43.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:10:43.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:10:43.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:10:43.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:10:43.006 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:10:43.006 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:10:43.006 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:10:43.006 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:10:43.006 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:10:43.006 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:10:43.006 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:10:43.006 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:10:43.006 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:10:43.006 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:10:43.006 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:10:43.006 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:10:43.006 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:10:43.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:10:43.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:10:43.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:10:43.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:10:43.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:10:43.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:10:43.008 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T02:10:43.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:10:43.008 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T02:10:43.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:10:43.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:10:43.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:10:43.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:10:43.023 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:43.012+0000 7f1e3ff3ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:10:43.024 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:43.012+0000 7f1e3ff3ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:10:43.025 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:43.012+0000 7f1e3ff3ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:10:43.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:10:43.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:10:43.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:10:43.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:10:43.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:10:43.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:10:43.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:10:43.165 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:10:43.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:10:43.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:10:43.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:10:43.397 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:43.384+0000 7f1e3ff3ba40 -1 Falling back to public interface 2026-02-13T02:10:44.025 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:44.012+0000 7f1e3ff3ba40 -1 osd.0 54 log_to_monitors true 2026-02-13T02:10:44.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:10:44.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:10:44.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:10:44.323 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:10:44.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:10:44.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:10:44.477 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 58 up_thru 58 down_at 55 last_clean_interval [43,54) [v2:127.0.0.1:6800/2647215216,v1:127.0.0.1:6801/2647215216] [v2:127.0.0.1:6802/2647215216,v1:127.0.0.1:6803/2647215216] exists,up e45fb68f-2f91-4f83-a0b4-08802390b8b5 2026-02-13T02:10:44.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:10:44.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:10:44.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:10:44.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:10:44.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:10:44.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:10:44.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:10:44.478 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:10:44.479 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:10:44.479 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:10:44.479 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:10:44.479 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:10:44.479 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:10:44.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:10:44.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:10:44.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:10:44.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:10:44.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:10:44.591 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:10:44.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:10:44.761 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:10:44.762 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:10:44.762 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:10:44.762 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:10:44.762 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:10:44.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=249108103170 2026-02-13T02:10:44.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 249108103170 2026-02-13T02:10:44.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-249108103170' 2026-02-13T02:10:44.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:10:44.838 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:10:44.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=219043332100 2026-02-13T02:10:44.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 219043332100 2026-02-13T02:10:44.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-249108103170 1-219043332100' 2026-02-13T02:10:44.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:10:44.909 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:10:44.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542157 2026-02-13T02:10:44.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542157 2026-02-13T02:10:44.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-249108103170 1-219043332100 2-60129542157' 2026-02-13T02:10:44.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:10:44.983 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-249108103170 2026-02-13T02:10:44.983 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:10:44.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:10:44.985 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-249108103170 2026-02-13T02:10:44.985 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:10:44.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=249108103170 2026-02-13T02:10:44.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 249108103170' 2026-02-13T02:10:44.987 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 249108103170 2026-02-13T02:10:44.987 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:10:45.144 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 249108103170 2026-02-13T02:10:45.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:10:46.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:10:46.146 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:10:46.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 249108103170 -lt 249108103170 2026-02-13T02:10:46.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:10:46.313 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-219043332100 2026-02-13T02:10:46.313 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:10:46.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:10:46.315 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-219043332100 2026-02-13T02:10:46.316 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:10:46.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=219043332100 2026-02-13T02:10:46.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 219043332100' 2026-02-13T02:10:46.317 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 219043332100 2026-02-13T02:10:46.317 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:10:46.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 219043332100 -lt 219043332100 2026-02-13T02:10:46.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:10:46.492 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542157 2026-02-13T02:10:46.492 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:10:46.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:10:46.494 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542157 2026-02-13T02:10:46.494 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:10:46.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542157 2026-02-13T02:10:46.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542157' 2026-02-13T02:10:46.496 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 60129542157 2026-02-13T02:10:46.496 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:10:46.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542157 -lt 60129542157 2026-02-13T02:10:46.657 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:10:46.657 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:10:46.657 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:10:46.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:10:46.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:10:46.857 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:10:46.857 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:10:46.857 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:10:46.857 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:10:46.858 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:10:46.858 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:10:47.016 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:10:47.016 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:10:47.017 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:10:47.017 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:10:47.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:10:47.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:10:47.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:10:47.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3784: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2026-02-13T02:10:47.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:10:47.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:10:47.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-13T02:10:47.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:10:47.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2026-02-13T02:10:47.217 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:10:47.217 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:10:47.217 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-13T02:10:47.217 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:10:47.217 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2026-02-13T02:10:47.217 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:10:47.217 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:10:47.217 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:10:47.217 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:10:47.217 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:10:47.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:10:47.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2026-02-13T02:10:47.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:10:47.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:10:47.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-13T02:10:47.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:10:47.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2026-02-13T02:10:47.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2026-02-13T02:10:47.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2026-02-13T02:10:47.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:10:47.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:10:47.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-13T02:10:47.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:10:47.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2026-02-13T02:10:47.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:10:47.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:10:47.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:10:47.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:10:47.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-13T02:10:47.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-13T02:10:47.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:10:47.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:10:47.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:10:47.830 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:10:47.830 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:10:47.830 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:10:47.830 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:10:47.830 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:10:47.830 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:10:47.831 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:10:47.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:10:47.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:10:47.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:10:47.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:10:47.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:10:47.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:10:47.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:10:47.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:10:47.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:10:47.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:10:47.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2026-02-13T02:10:47.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-13T02:10:47.833 INFO:tasks.workunit.client.0.trial112.stderr:start osd.2 2026-02-13T02:10:47.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:10:47.834 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2026-02-13T02:10:47.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-13T02:10:47.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:10:47.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:10:47.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:10:47.850 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:47.836+0000 7f413ce29a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:10:47.851 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:47.840+0000 7f413ce29a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:10:47.852 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:47.840+0000 7f413ce29a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:10:47.998 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-13T02:10:47.998 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:10:47.998 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T02:10:47.998 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:10:47.998 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:10:47.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:10:47.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:10:47.999 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:10:47.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:10:47.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:10:48.155 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:10:48.277 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:48.264+0000 7f413ce29a40 -1 Falling back to public interface 2026-02-13T02:10:48.905 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:48.892+0000 7f413ce29a40 -1 osd.2 60 log_to_monitors true 2026-02-13T02:10:49.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:10:49.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:10:49.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:10:49.157 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:10:49.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:10:49.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:10:49.324 INFO:tasks.workunit.client.0.trial112.stderr:osd.2 up in weight 1 up_from 63 up_thru 63 down_at 60 last_clean_interval [14,60) [v2:127.0.0.1:6818/3804626874,v1:127.0.0.1:6819/3804626874] [v2:127.0.0.1:6820/3804626874,v1:127.0.0.1:6821/3804626874] exists,up 92cde444-63c3-4cc0-ab5f-8698b24011e5 2026-02-13T02:10:49.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:10:49.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:10:49.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:10:49.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:10:49.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:10:49.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:10:49.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:10:49.325 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:10:49.326 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:10:49.326 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:10:49.326 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:10:49.326 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:10:49.326 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:10:49.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:10:49.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:10:49.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:10:49.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:10:49.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:10:49.429 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:10:49.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:10:49.593 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:10:49.593 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:10:49.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:10:49.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:10:49.593 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:10:49.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=249108103172 2026-02-13T02:10:49.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 249108103172 2026-02-13T02:10:49.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-249108103172' 2026-02-13T02:10:49.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:10:49.667 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:10:49.739 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=219043332102 2026-02-13T02:10:49.739 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 219043332102 2026-02-13T02:10:49.739 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-249108103172 1-219043332102' 2026-02-13T02:10:49.739 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:10:49.739 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:10:49.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=270582939650 2026-02-13T02:10:49.816 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 270582939650 2026-02-13T02:10:49.816 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-249108103172 1-219043332102 2-270582939650' 2026-02-13T02:10:49.816 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:10:49.816 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-249108103172 2026-02-13T02:10:49.817 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:10:49.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:10:49.818 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-249108103172 2026-02-13T02:10:49.819 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:10:49.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=249108103172 2026-02-13T02:10:49.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 249108103172' 2026-02-13T02:10:49.820 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 249108103172 2026-02-13T02:10:49.821 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:10:49.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 249108103170 -lt 249108103172 2026-02-13T02:10:49.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:10:50.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:10:50.994 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:10:51.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 249108103172 -lt 249108103172 2026-02-13T02:10:51.154 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:10:51.154 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-219043332102 2026-02-13T02:10:51.154 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:10:51.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:10:51.157 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-219043332102 2026-02-13T02:10:51.157 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:10:51.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=219043332102 2026-02-13T02:10:51.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 219043332102' 2026-02-13T02:10:51.158 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 219043332102 2026-02-13T02:10:51.159 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:10:51.336 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 219043332102 -lt 219043332102 2026-02-13T02:10:51.336 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:10:51.337 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-270582939650 2026-02-13T02:10:51.337 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:10:51.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:10:51.339 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-270582939650 2026-02-13T02:10:51.339 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:10:51.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=270582939650 2026-02-13T02:10:51.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 270582939650' 2026-02-13T02:10:51.341 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 270582939650 2026-02-13T02:10:51.341 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:10:51.509 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 270582939650 -lt 270582939650 2026-02-13T02:10:51.509 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:10:51.510 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:10:51.510 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:10:51.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:10:51.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:10:51.710 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:10:51.710 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:10:51.710 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:10:51.710 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:10:51.711 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:10:51.711 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:10:51.871 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:10:51.872 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:10:51.872 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:10:51.872 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:10:52.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:10:52.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:10:52.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:10:52.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3785: corrupt_scrub_erasure: echo -n val3-EOBJ4 2026-02-13T02:10:52.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3786: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2026-02-13T02:10:52.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:10:52.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:10:52.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-13T02:10:52.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:10:52.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2026-02-13T02:10:52.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:10:52.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:10:52.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-13T02:10:52.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:10:52.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2026-02-13T02:10:52.070 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:10:52.070 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:10:52.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:10:52.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:10:52.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:10:52.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:10:52.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2026-02-13T02:10:52.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:10:52.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:10:52.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-13T02:10:52.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:10:52.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2026-02-13T02:10:52.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2026-02-13T02:10:52.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2026-02-13T02:10:52.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:10:52.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:10:52.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-13T02:10:52.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:10:52.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2026-02-13T02:10:52.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:10:52.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:10:52.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:10:52.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:10:52.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-13T02:10:52.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-13T02:10:52.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:10:52.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:10:52.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:10:52.650 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:10:52.650 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:10:52.650 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:10:52.650 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:10:52.650 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:10:52.650 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:10:52.651 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:10:52.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:10:52.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:10:52.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:10:52.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:10:52.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:10:52.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:10:52.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:10:52.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:10:52.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:10:52.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:10:52.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2026-02-13T02:10:52.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-13T02:10:52.653 INFO:tasks.workunit.client.0.trial112.stderr:start osd.2 2026-02-13T02:10:52.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:10:52.653 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2026-02-13T02:10:52.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-13T02:10:52.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:10:52.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:10:52.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:10:52.670 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:52.656+0000 7f5bf8794a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:10:52.671 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:52.660+0000 7f5bf8794a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:10:52.672 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:52.660+0000 7f5bf8794a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:10:52.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-13T02:10:52.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:10:52.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T02:10:52.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:10:52.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:10:52.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:10:52.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:10:52.818 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:10:52.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:10:52.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:10:52.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:10:53.108 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:53.096+0000 7f5bf8794a40 -1 Falling back to public interface 2026-02-13T02:10:53.755 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:53.744+0000 7f5bf8794a40 -1 osd.2 64 log_to_monitors true 2026-02-13T02:10:53.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:10:53.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:10:53.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:10:53.985 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:10:53.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:10:53.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:10:54.148 INFO:tasks.workunit.client.0.trial112.stderr:osd.2 up in weight 1 up_from 68 up_thru 68 down_at 65 last_clean_interval [63,64) [v2:127.0.0.1:6818/3264290206,v1:127.0.0.1:6819/3264290206] [v2:127.0.0.1:6820/3264290206,v1:127.0.0.1:6821/3264290206] exists,up 92cde444-63c3-4cc0-ab5f-8698b24011e5 2026-02-13T02:10:54.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:10:54.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:10:54.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:10:54.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:10:54.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:10:54.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:10:54.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:10:54.149 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:10:54.150 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:10:54.150 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:10:54.150 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:10:54.150 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:10:54.150 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:10:54.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:10:54.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:10:54.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:10:54.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:10:54.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:10:54.262 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:10:54.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:10:54.421 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:10:54.422 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:10:54.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:10:54.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:10:54.422 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:10:54.498 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=249108103174 2026-02-13T02:10:54.498 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 249108103174 2026-02-13T02:10:54.498 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-249108103174' 2026-02-13T02:10:54.498 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:10:54.499 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:10:54.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=219043332104 2026-02-13T02:10:54.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 219043332104 2026-02-13T02:10:54.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-249108103174 1-219043332104' 2026-02-13T02:10:54.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:10:54.570 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:10:54.644 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=292057776130 2026-02-13T02:10:54.644 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 292057776130 2026-02-13T02:10:54.644 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-249108103174 1-219043332104 2-292057776130' 2026-02-13T02:10:54.644 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:10:54.644 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-249108103174 2026-02-13T02:10:54.645 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:10:54.646 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:10:54.646 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-249108103174 2026-02-13T02:10:54.647 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:10:54.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=249108103174 2026-02-13T02:10:54.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 249108103174' 2026-02-13T02:10:54.649 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 249108103174 2026-02-13T02:10:54.649 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:10:54.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 249108103173 -lt 249108103174 2026-02-13T02:10:54.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:10:55.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:10:55.819 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:10:55.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 249108103173 -lt 249108103174 2026-02-13T02:10:55.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:10:56.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:10:56.984 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:10:57.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 249108103174 -lt 249108103174 2026-02-13T02:10:57.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:10:57.142 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-219043332104 2026-02-13T02:10:57.143 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:10:57.144 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:10:57.145 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-219043332104 2026-02-13T02:10:57.145 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:10:57.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=219043332104 2026-02-13T02:10:57.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 219043332104' 2026-02-13T02:10:57.147 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 219043332104 2026-02-13T02:10:57.147 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:10:57.319 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 219043332104 -lt 219043332104 2026-02-13T02:10:57.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:10:57.320 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-292057776130 2026-02-13T02:10:57.320 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:10:57.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:10:57.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-292057776130 2026-02-13T02:10:57.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:10:57.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=292057776130 2026-02-13T02:10:57.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 292057776130' 2026-02-13T02:10:57.324 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 292057776130 2026-02-13T02:10:57.325 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:10:57.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 292057776130 -lt 292057776130 2026-02-13T02:10:57.496 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:10:57.496 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:10:57.496 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:10:57.694 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:10:57.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:10:57.696 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:10:57.696 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:10:57.696 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:10:57.696 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:10:57.696 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:10:57.696 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:10:57.859 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:10:57.860 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:10:57.860 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:10:57.860 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:10:58.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:10:58.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:10:58.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:10:58.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3787: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2026-02-13T02:10:58.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-13T02:10:58.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ5 2026-02-13T02:10:58.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2026-02-13T02:10:58.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:10:58.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-13T02:10:58.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ5 2026-02-13T02:10:58.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:10:58.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:10:58.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:10:58.273 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:10:58.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:10:58.477 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:10:58.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:10:58.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:10:58.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:10:58.516 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 5 % 2 2026-02-13T02:10:58.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=1 2026-02-13T02:10:58.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-13T02:10:58.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3792: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2026-02-13T02:10:58.519 INFO:tasks.workunit.client.0.trial112.stderr:2+0 records in 2026-02-13T02:10:58.519 INFO:tasks.workunit.client.0.trial112.stderr:2+0 records out 2026-02-13T02:10:58.519 INFO:tasks.workunit.client.0.trial112.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 9.5286e-05 s, 43.0 MB/s 2026-02-13T02:10:58.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3793: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-13T02:10:58.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:10:58.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:10:58.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:10:58.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:10:58.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-13T02:10:58.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:10:58.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:10:58.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:10:58.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:10:58.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:10:58.520 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:10:58.520 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:10:58.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:10:58.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:10:58.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:10:58.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:10:58.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-13T02:10:58.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:10:58.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:10:58.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:10:58.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:10:58.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:10:58.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-13T02:10:59.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-13T02:10:59.121 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:10:59.121 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:10:59.121 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:10:59.121 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:10:59.121 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:10:59.121 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:10:59.121 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:10:59.121 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:10:59.121 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:10:59.121 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:10:59.122 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:10:59.122 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:10:59.122 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:10:59.122 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:10:59.122 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:10:59.122 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:10:59.122 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:10:59.122 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:10:59.122 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:10:59.122 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:10:59.124 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:10:59.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:10:59.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:10:59.125 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:10:59.125 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:10:59.125 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:10:59.125 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:10:59.125 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:10:59.125 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:10:59.125 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:10:59.125 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:10:59.125 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:10:59.125 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:10:59.125 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:10:59.125 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:10:59.126 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:10:59.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:10:59.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:10:59.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:10:59.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:10:59.140 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:59.127+0000 7f5ebbd13a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:10:59.142 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:59.127+0000 7f5ebbd13a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:10:59.143 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:59.127+0000 7f5ebbd13a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:10:59.289 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:10:59.289 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:10:59.289 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:10:59.289 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:10:59.289 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:10:59.289 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:10:59.289 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:10:59.290 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:10:59.290 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:10:59.290 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:10:59.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:10:59.572 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:10:59.559+0000 7f5ebbd13a40 -1 Falling back to public interface 2026-02-13T02:11:00.197 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:00.183+0000 7f5ebbd13a40 -1 osd.1 71 log_to_monitors true 2026-02-13T02:11:00.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:11:00.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:11:00.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:11:00.447 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:11:00.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:11:00.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:11:00.606 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 75 up_thru 75 down_at 72 last_clean_interval [51,71) [v2:127.0.0.1:6810/151739231,v1:127.0.0.1:6811/151739231] [v2:127.0.0.1:6812/151739231,v1:127.0.0.1:6813/151739231] exists,up 396635b3-6e66-4635-9535-fe3bc01784cc 2026-02-13T02:11:00.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:11:00.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:11:00.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:11:00.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:11:00.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:11:00.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:11:00.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:11:00.607 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:11:00.608 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:11:00.608 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:11:00.608 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:11:00.609 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:11:00.609 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:11:00.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:11:00.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:11:00.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:11:00.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:11:00.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:11:00.714 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:11:00.876 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:11:00.876 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:11:00.877 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:11:00.877 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:11:00.877 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:11:00.877 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:11:00.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=249108103176 2026-02-13T02:11:00.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 249108103176 2026-02-13T02:11:00.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-249108103176' 2026-02-13T02:11:00.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:11:00.952 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:11:01.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=322122547202 2026-02-13T02:11:01.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 322122547202 2026-02-13T02:11:01.023 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-249108103176 1-322122547202' 2026-02-13T02:11:01.023 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:11:01.023 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:11:01.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=292057776132 2026-02-13T02:11:01.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 292057776132 2026-02-13T02:11:01.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-249108103176 1-322122547202 2-292057776132' 2026-02-13T02:11:01.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:11:01.097 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-249108103176 2026-02-13T02:11:01.097 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:11:01.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:11:01.099 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-249108103176 2026-02-13T02:11:01.099 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:11:01.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=249108103176 2026-02-13T02:11:01.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 249108103176' 2026-02-13T02:11:01.101 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 249108103176 2026-02-13T02:11:01.101 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:11:01.267 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 249108103175 -lt 249108103176 2026-02-13T02:11:01.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:11:02.269 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:11:02.269 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:11:02.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 249108103176 -lt 249108103176 2026-02-13T02:11:02.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:11:02.432 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-322122547202 2026-02-13T02:11:02.432 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:11:02.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:11:02.434 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-322122547202 2026-02-13T02:11:02.434 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:11:02.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=322122547202 2026-02-13T02:11:02.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 322122547202' 2026-02-13T02:11:02.436 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 322122547202 2026-02-13T02:11:02.436 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:11:02.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 322122547202 -lt 322122547202 2026-02-13T02:11:02.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:11:02.608 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-292057776132 2026-02-13T02:11:02.608 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:11:02.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:11:02.610 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-292057776132 2026-02-13T02:11:02.610 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:11:02.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=292057776132 2026-02-13T02:11:02.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 292057776132' 2026-02-13T02:11:02.612 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 292057776132 2026-02-13T02:11:02.612 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:11:02.774 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 292057776132 -lt 292057776132 2026-02-13T02:11:02.775 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:11:02.775 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:11:02.775 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:11:02.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:11:02.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:11:02.969 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:11:02.969 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:11:02.969 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:11:02.969 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:11:02.969 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:11:02.969 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:11:03.121 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:11:03.122 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:11:03.122 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:11:03.122 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:11:03.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:11:03.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:11:03.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:11:03.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-13T02:11:03.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ6 2026-02-13T02:11:03.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2026-02-13T02:11:03.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:11:03.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-13T02:11:03.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ6 2026-02-13T02:11:03.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:11:03.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:11:03.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:11:03.517 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:11:03.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:11:03.723 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:11:03.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:11:03.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:11:03.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:11:03.754 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 6 % 2 2026-02-13T02:11:03.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=0 2026-02-13T02:11:03.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-13T02:11:03.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3797: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2026-02-13T02:11:03.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:11:03.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:11:03.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T02:11:03.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:11:03.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2026-02-13T02:11:03.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:11:03.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:11:03.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T02:11:03.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:11:03.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T02:11:03.758 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:11:03.758 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:11:03.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:11:03.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:11:03.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:11:04.066 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:11:04.066 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2026-02-13T02:11:04.066 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:11:04.066 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:11:04.066 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:11:04.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:11:04.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:11:04.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2026-02-13T02:11:04.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-13T02:11:04.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:11:04.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:11:04.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:11:04.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:11:04.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:11:04.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:11:04.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:11:04.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:11:04.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:11:04.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:11:04.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:11:04.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:11:04.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:11:04.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:11:04.341 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:11:04.341 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:11:04.342 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:11:04.342 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:11:04.342 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:11:04.342 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:11:04.342 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:11:04.343 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:11:04.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:11:04.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:11:04.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:11:04.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:11:04.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:11:04.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:11:04.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:11:04.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:11:04.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:11:04.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:11:04.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:11:04.345 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T02:11:04.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:11:04.345 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T02:11:04.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:11:04.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:11:04.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:11:04.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:11:04.359 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:04.347+0000 7f69833e8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:11:04.361 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:04.347+0000 7f69833e8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:11:04.361 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:04.347+0000 7f69833e8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:11:04.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:11:04.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:11:04.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:11:04.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:11:04.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:11:04.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:11:04.502 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:11:04.502 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:11:04.502 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:11:04.502 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:11:04.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:11:04.777 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:04.763+0000 7f69833e8a40 -1 Falling back to public interface 2026-02-13T02:11:05.355 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:05.339+0000 7f69833e8a40 -1 osd.0 78 log_to_monitors true 2026-02-13T02:11:05.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:11:05.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:11:05.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:11:05.667 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:11:05.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:11:05.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:11:05.828 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 82 up_thru 82 down_at 79 last_clean_interval [58,78) [v2:127.0.0.1:6800/2632664882,v1:127.0.0.1:6801/2632664882] [v2:127.0.0.1:6802/2632664882,v1:127.0.0.1:6803/2632664882] exists,up e45fb68f-2f91-4f83-a0b4-08802390b8b5 2026-02-13T02:11:05.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:11:05.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:11:05.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:11:05.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:11:05.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:11:05.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:11:05.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:11:05.829 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:11:05.830 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:11:05.830 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:11:05.830 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:11:05.830 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:11:05.830 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:11:05.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:11:05.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:11:05.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:11:05.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:11:05.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:11:05.939 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:11:06.104 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:11:06.104 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:11:06.104 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:11:06.104 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:11:06.104 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:11:06.105 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:11:06.183 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=352187318274 2026-02-13T02:11:06.184 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 352187318274 2026-02-13T02:11:06.184 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-352187318274' 2026-02-13T02:11:06.184 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:11:06.184 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:11:06.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=322122547204 2026-02-13T02:11:06.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 322122547204 2026-02-13T02:11:06.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-352187318274 1-322122547204' 2026-02-13T02:11:06.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:11:06.257 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:11:06.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=292057776134 2026-02-13T02:11:06.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 292057776134 2026-02-13T02:11:06.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-352187318274 1-322122547204 2-292057776134' 2026-02-13T02:11:06.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:11:06.330 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-352187318274 2026-02-13T02:11:06.331 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:11:06.331 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:11:06.332 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-352187318274 2026-02-13T02:11:06.332 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:11:06.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=352187318274 2026-02-13T02:11:06.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 352187318274' 2026-02-13T02:11:06.334 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 352187318274 2026-02-13T02:11:06.334 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:11:06.505 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 352187318274 -lt 352187318274 2026-02-13T02:11:06.506 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:11:06.506 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-322122547204 2026-02-13T02:11:06.506 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:11:06.508 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:11:06.508 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-322122547204 2026-02-13T02:11:06.509 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:11:06.510 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=322122547204 2026-02-13T02:11:06.510 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 322122547204' 2026-02-13T02:11:06.510 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 322122547204 2026-02-13T02:11:06.511 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:11:06.680 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 322122547203 -lt 322122547204 2026-02-13T02:11:06.680 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:11:07.681 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:11:07.682 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:11:07.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 322122547203 -lt 322122547204 2026-02-13T02:11:07.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:11:08.847 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:11:08.848 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:11:09.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 322122547204 -lt 322122547204 2026-02-13T02:11:09.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:11:09.011 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-292057776134 2026-02-13T02:11:09.011 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:11:09.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:11:09.013 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-292057776134 2026-02-13T02:11:09.013 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:11:09.015 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=292057776134 2026-02-13T02:11:09.015 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 292057776134' 2026-02-13T02:11:09.015 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 292057776134 2026-02-13T02:11:09.015 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:11:09.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 292057776134 -lt 292057776134 2026-02-13T02:11:09.191 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:11:09.192 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:11:09.192 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:11:09.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:11:09.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:11:09.392 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:11:09.392 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:11:09.392 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:11:09.392 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:11:09.392 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:11:09.392 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:11:09.556 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:11:09.556 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:11:09.557 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:11:09.557 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:11:09.762 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:11:09.762 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:11:09.762 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:11:09.762 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3798: corrupt_scrub_erasure: echo -n bad-val 2026-02-13T02:11:09.762 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3799: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2026-02-13T02:11:09.762 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:11:09.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:11:09.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:11:09.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:11:09.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2026-02-13T02:11:09.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:11:09.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:11:09.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:11:09.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:11:09.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:11:09.763 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:11:09.763 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:11:09.764 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:11:09.764 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:11:09.764 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:11:10.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:11:10.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2026-02-13T02:11:10.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:11:10.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:11:10.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:11:10.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:11:10.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:11:10.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2026-02-13T02:11:10.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-13T02:11:10.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:11:10.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:11:10.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:11:10.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:11:10.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:11:10.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:11:10.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:11:10.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:11:10.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:11:10.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:11:10.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:11:10.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:11:10.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:11:10.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:11:10.346 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:11:10.346 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:11:10.346 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:11:10.346 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:11:10.347 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:11:10.347 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:11:10.347 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:11:10.348 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:11:10.348 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:11:10.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:11:10.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:11:10.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:11:10.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:11:10.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:11:10.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:11:10.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:11:10.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:11:10.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:11:10.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:11:10.349 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:11:10.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:11:10.350 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:11:10.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:11:10.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:11:10.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:11:10.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:11:10.366 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:10.351+0000 7f1301159a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:11:10.368 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:10.355+0000 7f1301159a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:11:10.369 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:10.355+0000 7f1301159a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:11:10.513 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:11:10.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:11:10.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:11:10.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:11:10.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:11:10.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:11:10.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:11:10.514 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:11:10.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:11:10.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:11:10.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:11:10.813 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:10.799+0000 7f1301159a40 -1 Falling back to public interface 2026-02-13T02:11:11.441 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:11.427+0000 7f1301159a40 -1 osd.1 83 log_to_monitors true 2026-02-13T02:11:11.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:11:11.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:11:11.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:11:11.671 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:11:11.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:11:11.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:11:11.834 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 87 up_thru 87 down_at 84 last_clean_interval [75,83) [v2:127.0.0.1:6810/832670782,v1:127.0.0.1:6811/832670782] [v2:127.0.0.1:6812/832670782,v1:127.0.0.1:6813/832670782] exists,up 396635b3-6e66-4635-9535-fe3bc01784cc 2026-02-13T02:11:11.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:11:11.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:11:11.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:11:11.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:11:11.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:11:11.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:11:11.835 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:11:11.835 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:11:11.835 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:11:11.836 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:11:11.836 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:11:11.836 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:11:11.836 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:11:11.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:11:11.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:11:11.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:11:11.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:11:11.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:11:11.948 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:11:12.108 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:11:12.108 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:11:12.108 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:11:12.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:11:12.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:11:12.109 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:11:12.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=352187318276 2026-02-13T02:11:12.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 352187318276 2026-02-13T02:11:12.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-352187318276' 2026-02-13T02:11:12.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:11:12.186 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:11:12.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=373662154754 2026-02-13T02:11:12.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 373662154754 2026-02-13T02:11:12.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-352187318276 1-373662154754' 2026-02-13T02:11:12.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:11:12.260 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:11:12.331 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=292057776136 2026-02-13T02:11:12.331 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 292057776136 2026-02-13T02:11:12.332 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-352187318276 1-373662154754 2-292057776136' 2026-02-13T02:11:12.332 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:11:12.332 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-352187318276 2026-02-13T02:11:12.332 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:11:12.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:11:12.334 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-352187318276 2026-02-13T02:11:12.334 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:11:12.336 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=352187318276 2026-02-13T02:11:12.336 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 352187318276' 2026-02-13T02:11:12.336 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 352187318276 2026-02-13T02:11:12.336 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:11:12.498 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 352187318276 -lt 352187318276 2026-02-13T02:11:12.498 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:11:12.499 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-373662154754 2026-02-13T02:11:12.499 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:11:12.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:11:12.501 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-373662154754 2026-02-13T02:11:12.501 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:11:12.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=373662154754 2026-02-13T02:11:12.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 373662154754' 2026-02-13T02:11:12.503 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 373662154754 2026-02-13T02:11:12.504 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:11:12.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 373662154754 2026-02-13T02:11:12.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:11:13.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:11:13.672 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:11:13.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 373662154754 2026-02-13T02:11:13.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:11:14.839 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:11:14.840 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:11:15.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 373662154754 -lt 373662154754 2026-02-13T02:11:15.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:11:15.002 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-292057776136 2026-02-13T02:11:15.003 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:11:15.004 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:11:15.005 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-292057776136 2026-02-13T02:11:15.005 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:11:15.006 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=292057776136 2026-02-13T02:11:15.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 292057776136' 2026-02-13T02:11:15.007 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 292057776136 2026-02-13T02:11:15.007 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:11:15.181 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 292057776137 -lt 292057776136 2026-02-13T02:11:15.181 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:11:15.182 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:11:15.182 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:11:15.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:11:15.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:11:15.373 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:11:15.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:11:15.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:11:15.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:11:15.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:11:15.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:11:15.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:11:15.529 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:11:15.529 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:11:15.529 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:11:15.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:11:15.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:11:15.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:11:15.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-13T02:11:15.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ7 2026-02-13T02:11:15.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2026-02-13T02:11:15.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:11:15.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-13T02:11:15.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ7 2026-02-13T02:11:15.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:11:15.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:11:15.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:11:15.929 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:11:15.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:11:16.133 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:11:16.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:11:16.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:11:16.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:11:16.168 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 7 % 2 2026-02-13T02:11:16.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=1 2026-02-13T02:11:16.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-13T02:11:16.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3803: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2026-02-13T02:11:16.170 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3804: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2026-02-13T02:11:16.170 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3805: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2026-02-13T02:11:16.190 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3808: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2026-02-13T02:11:16.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:11:16.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:11:16.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T02:11:16.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:11:16.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2026-02-13T02:11:16.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:11:16.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:11:16.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T02:11:16.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:11:16.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T02:11:16.192 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:11:16.192 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:11:16.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:11:16.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:11:16.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:11:16.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:11:16.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2026-02-13T02:11:16.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:11:16.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:11:16.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:11:16.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:11:16.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:11:16.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2026-02-13T02:11:16.646 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-13T02:11:16.646 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:11:16.646 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:11:16.646 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:11:16.646 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:11:16.646 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:11:16.646 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:11:16.646 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:11:16.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:11:16.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:11:16.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:11:16.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:11:16.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:11:16.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:11:16.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:11:16.647 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:11:16.647 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:11:16.647 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:11:16.647 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:11:16.647 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:11:16.648 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:11:16.648 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:11:16.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:11:16.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:11:16.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:11:16.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:11:16.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:11:16.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:11:16.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:11:16.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:11:16.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:11:16.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:11:16.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:11:16.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:11:16.650 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T02:11:16.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:11:16.650 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T02:11:16.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:11:16.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:11:16.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:11:16.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:11:16.665 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:16.651+0000 7f0f45eaea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:11:16.666 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:16.651+0000 7f0f45eaea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:11:16.667 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:16.655+0000 7f0f45eaea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:11:16.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:11:16.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:11:16.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:11:16.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:11:16.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:11:16.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:11:16.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:11:16.812 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:11:16.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:11:16.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:11:16.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:11:17.073 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:17.059+0000 7f0f45eaea40 -1 Falling back to public interface 2026-02-13T02:11:17.646 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:17.631+0000 7f0f45eaea40 -1 osd.0 90 log_to_monitors true 2026-02-13T02:11:17.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:11:17.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:11:17.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:11:17.979 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:11:17.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:11:17.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:11:18.133 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 94 up_thru 94 down_at 91 last_clean_interval [82,90) [v2:127.0.0.1:6800/3768527895,v1:127.0.0.1:6801/3768527895] [v2:127.0.0.1:6802/3768527895,v1:127.0.0.1:6803/3768527895] exists,up e45fb68f-2f91-4f83-a0b4-08802390b8b5 2026-02-13T02:11:18.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:11:18.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:11:18.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:11:18.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:11:18.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:11:18.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:11:18.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:11:18.134 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:11:18.135 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:11:18.135 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:11:18.135 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:11:18.135 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:11:18.135 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:11:18.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:11:18.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:11:18.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:11:18.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:11:18.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:11:18.247 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:11:18.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:11:18.406 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:11:18.407 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:11:18.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:11:18.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:11:18.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:11:18.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=403726925826 2026-02-13T02:11:18.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 403726925826 2026-02-13T02:11:18.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-403726925826' 2026-02-13T02:11:18.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:11:18.480 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:11:18.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=373662154756 2026-02-13T02:11:18.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 373662154756 2026-02-13T02:11:18.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-403726925826 1-373662154756' 2026-02-13T02:11:18.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:11:18.551 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:11:18.624 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=292057776138 2026-02-13T02:11:18.624 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 292057776138 2026-02-13T02:11:18.624 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-403726925826 1-373662154756 2-292057776138' 2026-02-13T02:11:18.624 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:11:18.625 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-403726925826 2026-02-13T02:11:18.625 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:11:18.626 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:11:18.627 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-403726925826 2026-02-13T02:11:18.627 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:11:18.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=403726925826 2026-02-13T02:11:18.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 403726925826' 2026-02-13T02:11:18.629 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 403726925826 2026-02-13T02:11:18.629 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:11:18.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 403726925826 2026-02-13T02:11:18.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:11:19.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:11:19.794 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:11:19.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 403726925826 2026-02-13T02:11:19.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:11:20.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:11:20.964 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:11:21.128 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 403726925826 -lt 403726925826 2026-02-13T02:11:21.128 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:11:21.129 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-373662154756 2026-02-13T02:11:21.129 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:11:21.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:11:21.131 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-373662154756 2026-02-13T02:11:21.132 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:11:21.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=373662154756 2026-02-13T02:11:21.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 373662154756' 2026-02-13T02:11:21.133 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 373662154756 2026-02-13T02:11:21.134 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:11:21.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 373662154756 -lt 373662154756 2026-02-13T02:11:21.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:11:21.303 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-292057776138 2026-02-13T02:11:21.303 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:11:21.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:11:21.305 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-292057776138 2026-02-13T02:11:21.305 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:11:21.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=292057776138 2026-02-13T02:11:21.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 292057776138' 2026-02-13T02:11:21.307 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 292057776138 2026-02-13T02:11:21.307 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:11:21.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 292057776139 -lt 292057776138 2026-02-13T02:11:21.484 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:11:21.485 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:11:21.485 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:11:21.681 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:11:21.681 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:11:21.682 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:11:21.682 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:11:21.682 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:11:21.683 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:11:21.683 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:11:21.683 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:11:21.836 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:11:21.837 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:11:21.837 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:11:21.837 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:11:22.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:11:22.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:11:22.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:11:22.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3809: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2026-02-13T02:11:22.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:11:22.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:11:22.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T02:11:22.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:11:22.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2026-02-13T02:11:22.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:11:22.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:11:22.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T02:11:22.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:11:22.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T02:11:22.026 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:11:22.026 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:11:22.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:11:22.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:11:22.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:11:22.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:11:22.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2026-02-13T02:11:22.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:11:22.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:11:22.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:11:22.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:11:22.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:11:22.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2026-02-13T02:11:22.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-13T02:11:22.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:11:22.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:11:22.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:11:22.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:11:22.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:11:22.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:11:22.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:11:22.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:11:22.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:11:22.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:11:22.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:11:22.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:11:22.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:11:22.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:11:22.631 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:11:22.631 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:11:22.631 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:11:22.633 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:11:22.633 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:11:22.633 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:11:22.633 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:11:22.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:11:22.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:11:22.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:11:22.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:11:22.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:11:22.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:11:22.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:11:22.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:11:22.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:11:22.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:11:22.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:11:22.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:11:22.634 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T02:11:22.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:11:22.635 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T02:11:22.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:11:22.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:11:22.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:11:22.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:11:22.656 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:22.643+0000 7f9dcbdfea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:11:22.657 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:22.643+0000 7f9dcbdfea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:11:22.658 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:22.643+0000 7f9dcbdfea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:11:22.795 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:11:22.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:11:22.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:11:22.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:11:22.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:11:22.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:11:22.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:11:22.796 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:11:22.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:11:22.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:11:22.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:11:23.041 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:23.027+0000 7f9dcbdfea40 -1 Falling back to public interface 2026-02-13T02:11:23.600 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:23.587+0000 7f9dcbdfea40 -1 osd.0 95 log_to_monitors true 2026-02-13T02:11:23.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:11:23.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:11:23.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:11:23.956 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:11:23.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:11:23.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:11:24.115 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 99 up_thru 99 down_at 96 last_clean_interval [94,95) [v2:127.0.0.1:6800/1586325175,v1:127.0.0.1:6801/1586325175] [v2:127.0.0.1:6802/1586325175,v1:127.0.0.1:6803/1586325175] exists,up e45fb68f-2f91-4f83-a0b4-08802390b8b5 2026-02-13T02:11:24.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:11:24.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:11:24.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:11:24.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:11:24.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:11:24.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:11:24.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:11:24.117 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:11:24.117 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:11:24.117 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:11:24.117 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:11:24.117 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:11:24.118 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:11:24.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:11:24.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:11:24.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:11:24.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:11:24.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:11:24.226 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:11:24.395 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:11:24.395 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:11:24.395 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:11:24.395 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:11:24.395 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:11:24.395 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:11:24.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=425201762306 2026-02-13T02:11:24.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 425201762306 2026-02-13T02:11:24.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-425201762306' 2026-02-13T02:11:24.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:11:24.467 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:11:24.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=373662154758 2026-02-13T02:11:24.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 373662154758 2026-02-13T02:11:24.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-425201762306 1-373662154758' 2026-02-13T02:11:24.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:11:24.540 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:11:24.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=292057776141 2026-02-13T02:11:24.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 292057776141 2026-02-13T02:11:24.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-425201762306 1-373662154758 2-292057776141' 2026-02-13T02:11:24.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:11:24.614 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-425201762306 2026-02-13T02:11:24.615 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:11:24.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:11:24.617 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-425201762306 2026-02-13T02:11:24.617 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:11:24.618 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=425201762306 2026-02-13T02:11:24.618 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 425201762306' 2026-02-13T02:11:24.618 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 425201762306 2026-02-13T02:11:24.619 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:11:24.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 425201762306 2026-02-13T02:11:24.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:11:25.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:11:25.791 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:11:25.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 425201762306 2026-02-13T02:11:25.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:11:26.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:11:26.954 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:11:27.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 425201762306 -lt 425201762306 2026-02-13T02:11:27.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:11:27.118 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-373662154758 2026-02-13T02:11:27.118 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:11:27.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:11:27.120 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-373662154758 2026-02-13T02:11:27.120 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:11:27.122 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=373662154758 2026-02-13T02:11:27.122 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 373662154758' 2026-02-13T02:11:27.122 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 373662154758 2026-02-13T02:11:27.122 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:11:27.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 373662154758 -lt 373662154758 2026-02-13T02:11:27.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:11:27.281 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-292057776141 2026-02-13T02:11:27.281 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:11:27.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:11:27.284 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-292057776141 2026-02-13T02:11:27.284 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:11:27.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=292057776141 2026-02-13T02:11:27.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 292057776141' 2026-02-13T02:11:27.286 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 292057776141 2026-02-13T02:11:27.286 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:11:27.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 292057776141 -lt 292057776141 2026-02-13T02:11:27.461 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:11:27.462 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:11:27.462 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:11:27.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:11:27.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:11:27.660 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:11:27.660 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:11:27.661 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:11:27.661 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:11:27.661 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:11:27.661 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:11:27.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:11:27.819 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:11:27.819 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:11:27.819 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:11:28.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:11:28.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:11:28.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:11:28.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3810: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2026-02-13T02:11:28.016 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3816: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2026-02-13T02:11:28.016 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-13T02:11:28.016 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=EOBJ0 2026-02-13T02:11:28.016 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool EOBJ0 2026-02-13T02:11:28.016 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T02:11:28.176 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3816: corrupt_scrub_erasure: local pg=3.0 2026-02-13T02:11:28.176 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3818: corrupt_scrub_erasure: pg_scrub 3.0 2026-02-13T02:11:28.176 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=3.0 2026-02-13T02:11:28.176 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 3.0 2026-02-13T02:11:28.176 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-13T02:11:28.176 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-13T02:11:28.177 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:11:28.177 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:11:28.177 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:11:28.178 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:11:28.178 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:11:28.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-13T02:11:28.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-13T02:11:28.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-13T02:11:28.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-13T02:11:28.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:11:28.461 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:11:28.626 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:11:28.627 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:11:28.627 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:11:28.627 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:11:28.627 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:11:28.627 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:11:28.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=425201762308 2026-02-13T02:11:28.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 425201762308 2026-02-13T02:11:28.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-425201762308' 2026-02-13T02:11:28.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:11:28.702 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:11:28.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=373662154760 2026-02-13T02:11:28.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 373662154760 2026-02-13T02:11:28.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-425201762308 1-373662154760' 2026-02-13T02:11:28.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:11:28.777 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:11:28.848 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=292057776142 2026-02-13T02:11:28.848 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 292057776142 2026-02-13T02:11:28.848 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-425201762308 1-373662154760 2-292057776142' 2026-02-13T02:11:28.848 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:11:28.849 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-425201762308 2026-02-13T02:11:28.849 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:11:28.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:11:28.851 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-425201762308 2026-02-13T02:11:28.851 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:11:28.852 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=425201762308 2026-02-13T02:11:28.852 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 425201762308' 2026-02-13T02:11:28.853 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 425201762308 2026-02-13T02:11:28.853 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:11:29.017 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 425201762306 -lt 425201762308 2026-02-13T02:11:29.017 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:11:30.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:11:30.019 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:11:30.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 425201762306 -lt 425201762308 2026-02-13T02:11:30.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:11:31.180 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:11:31.181 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:11:31.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 425201762308 -lt 425201762308 2026-02-13T02:11:31.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:11:31.346 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-373662154760 2026-02-13T02:11:31.347 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:11:31.348 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:11:31.349 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-373662154760 2026-02-13T02:11:31.349 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:11:31.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=373662154760 2026-02-13T02:11:31.350 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 373662154760 2026-02-13T02:11:31.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 373662154760' 2026-02-13T02:11:31.351 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:11:31.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 373662154760 -lt 373662154760 2026-02-13T02:11:31.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:11:31.525 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-292057776142 2026-02-13T02:11:31.525 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:11:31.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:11:31.527 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-292057776142 2026-02-13T02:11:31.527 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:11:31.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=292057776142 2026-02-13T02:11:31.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 292057776142' 2026-02-13T02:11:31.529 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 292057776142 2026-02-13T02:11:31.529 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:11:31.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 292057776143 -lt 292057776142 2026-02-13T02:11:31.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-13T02:11:31.698 INFO:tasks.workunit.client.0.trial112.stdout:#---------- 3.0 loop 0 2026-02-13T02:11:31.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-13T02:11:31.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-13T02:11:31.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-13T02:11:31.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-13T02:11:31.699 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-13T02:11:31.699 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-13T02:11:31.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-13T02:11:31.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-13T02:11:31.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-13T02:11:31.773 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-13T02:11:31.773 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: get_last_scrub_stamp 3.0 2026-02-13T02:11:31.773 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:11:31.773 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:11:31.773 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:11:31.773 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:11:31.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: local last_scrub=2026-02-13T02:10:16.209170+0000 2026-02-13T02:11:31.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_scrub: ceph pg scrub 3.0 2026-02-13T02:11:32.089 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 3.0s0 on osd.1 to scrub 2026-02-13T02:11:32.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_scrub: wait_for_scrub 3.0 2026-02-13T02:10:16.209170+0000 2026-02-13T02:11:32.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-13T02:11:32.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:10:16.209170+0000 2026-02-13T02:11:32.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:11:32.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:11:32.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:11:32.102 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-13T02:11:32.102 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:11:32.102 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:11:32.102 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:11:32.102 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:11:32.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:10:16.209170+0000 '>' 2026-02-13T02:10:16.209170+0000 2026-02-13T02:11:32.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:11:33.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:11:33.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:11:33.264 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-13T02:11:33.264 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:11:33.264 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:11:33.264 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:11:33.265 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:11:33.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:10:16.209170+0000 '>' 2026-02-13T02:10:16.209170+0000 2026-02-13T02:11:33.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:11:34.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:11:34.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:11:34.430 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-13T02:11:34.430 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:11:34.430 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:11:34.430 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:11:34.431 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:11:34.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:10:16.209170+0000 '>' 2026-02-13T02:10:16.209170+0000 2026-02-13T02:11:34.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:11:35.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:11:35.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:11:35.592 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-13T02:11:35.592 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:11:35.592 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:11:35.592 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:11:35.592 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:11:35.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:10:16.209170+0000 '>' 2026-02-13T02:10:16.209170+0000 2026-02-13T02:11:35.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:11:36.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:11:36.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:11:36.755 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-13T02:11:36.755 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:11:36.755 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:11:36.756 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:11:36.756 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:11:36.914 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:10:16.209170+0000 '>' 2026-02-13T02:10:16.209170+0000 2026-02-13T02:11:36.915 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:11:37.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:11:37.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:11:37.917 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-13T02:11:37.917 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:11:37.917 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:11:37.918 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:11:37.918 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:11:38.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:10:16.209170+0000 '>' 2026-02-13T02:10:16.209170+0000 2026-02-13T02:11:38.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:11:39.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:11:39.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:11:39.079 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-13T02:11:39.079 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:11:39.079 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:11:39.079 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:11:39.080 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:11:39.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:11:32.474266+0000 '>' 2026-02-13T02:10:16.209170+0000 2026-02-13T02:11:39.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:11:39.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3820: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2026-02-13T02:11:39.256 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3822: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2026-02-13T02:11:39.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3822: corrupt_scrub_erasure: test 1 = 1 2026-02-13T02:11:39.269 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3824: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2026-02-13T02:11:39.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3824: corrupt_scrub_erasure: test 3.0 = 3.0 2026-02-13T02:11:39.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3826: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2026-02-13T02:11:39.296 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3828: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2026-02-13T02:11:39.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3828: corrupt_scrub_erasure: epoch=99 2026-02-13T02:11:39.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3830: corrupt_scrub_erasure: jq 'def walk(f): 2026-02-13T02:11:39.306 INFO:tasks.workunit.client.0.trial112.stderr: . as $in 2026-02-13T02:11:39.307 INFO:tasks.workunit.client.0.trial112.stderr: | if type == "object" then 2026-02-13T02:11:39.307 INFO:tasks.workunit.client.0.trial112.stderr: reduce keys[] as $key 2026-02-13T02:11:39.307 INFO:tasks.workunit.client.0.trial112.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-13T02:11:39.307 INFO:tasks.workunit.client.0.trial112.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-13T02:11:39.307 INFO:tasks.workunit.client.0.trial112.stderr: else f 2026-02-13T02:11:39.307 INFO:tasks.workunit.client.0.trial112.stderr: end; 2026-02-13T02:11:39.307 INFO:tasks.workunit.client.0.trial112.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-13T02:11:39.307 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-13T02:11:39.307 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-13T02:11:39.307 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-13T02:11:39.308 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-13T02:11:39.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3830: corrupt_scrub_erasure: jq .inconsistents 2026-02-13T02:11:39.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3830: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-13T02:11:39.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4458: corrupt_scrub_erasure: jq 'def walk(f): 2026-02-13T02:11:39.323 INFO:tasks.workunit.client.0.trial112.stderr: . as $in 2026-02-13T02:11:39.323 INFO:tasks.workunit.client.0.trial112.stderr: | if type == "object" then 2026-02-13T02:11:39.323 INFO:tasks.workunit.client.0.trial112.stderr: reduce keys[] as $key 2026-02-13T02:11:39.323 INFO:tasks.workunit.client.0.trial112.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-13T02:11:39.323 INFO:tasks.workunit.client.0.trial112.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-13T02:11:39.323 INFO:tasks.workunit.client.0.trial112.stderr: else f 2026-02-13T02:11:39.323 INFO:tasks.workunit.client.0.trial112.stderr: end; 2026-02-13T02:11:39.323 INFO:tasks.workunit.client.0.trial112.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-13T02:11:39.324 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-13T02:11:39.324 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-13T02:11:39.324 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-13T02:11:39.324 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2026-02-13T02:11:39.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4458: corrupt_scrub_erasure: jq .inconsistents 2026-02-13T02:11:39.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4458: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-13T02:11:39.348 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4459: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-13T02:11:39.348 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2513: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-13T02:11:39.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4460: corrupt_scrub_erasure: test no = yes 2026-02-13T02:11:39.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4465: corrupt_scrub_erasure: test '' = yes 2026-02-13T02:11:39.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4470: corrupt_scrub_erasure: pg_deep_scrub 3.0 2026-02-13T02:11:39.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1932: pg_deep_scrub: local pgid=3.0 2026-02-13T02:11:39.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: pg_deep_scrub: wait_for_pg_clean 3.0 2026-02-13T02:11:39.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-13T02:11:39.350 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-13T02:11:39.351 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:11:39.351 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:11:39.351 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:11:39.351 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:11:39.351 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:11:39.646 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-13T02:11:39.646 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-13T02:11:39.646 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-13T02:11:39.646 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-13T02:11:39.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:11:39.647 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:11:39.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:11:39.804 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:11:39.804 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:11:39.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:11:39.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:11:39.804 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:11:39.878 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=425201762311 2026-02-13T02:11:39.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 425201762311 2026-02-13T02:11:39.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-425201762311' 2026-02-13T02:11:39.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:11:39.879 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:11:39.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=373662154763 2026-02-13T02:11:39.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 373662154763 2026-02-13T02:11:39.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-425201762311 1-373662154763' 2026-02-13T02:11:39.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:11:39.952 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:11:40.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=292057776146 2026-02-13T02:11:40.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 292057776146 2026-02-13T02:11:40.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-425201762311 1-373662154763 2-292057776146' 2026-02-13T02:11:40.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:11:40.027 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-425201762311 2026-02-13T02:11:40.027 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:11:40.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:11:40.029 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-425201762311 2026-02-13T02:11:40.029 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:11:40.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=425201762311 2026-02-13T02:11:40.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 425201762311' 2026-02-13T02:11:40.031 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 425201762311 2026-02-13T02:11:40.031 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:11:40.203 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 425201762309 -lt 425201762311 2026-02-13T02:11:40.203 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:11:41.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:11:41.205 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:11:41.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 425201762311 -lt 425201762311 2026-02-13T02:11:41.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:11:41.366 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-373662154763 2026-02-13T02:11:41.366 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:11:41.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:11:41.368 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-373662154763 2026-02-13T02:11:41.368 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:11:41.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=373662154763 2026-02-13T02:11:41.370 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 373662154763 2026-02-13T02:11:41.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 373662154763' 2026-02-13T02:11:41.371 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:11:41.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 373662154763 -lt 373662154763 2026-02-13T02:11:41.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:11:41.541 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-292057776146 2026-02-13T02:11:41.541 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:11:41.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:11:41.543 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-292057776146 2026-02-13T02:11:41.543 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:11:41.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=292057776146 2026-02-13T02:11:41.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 292057776146' 2026-02-13T02:11:41.545 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 292057776146 2026-02-13T02:11:41.545 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:11:41.704 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 292057776146 -lt 292057776146 2026-02-13T02:11:41.704 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-13T02:11:41.705 INFO:tasks.workunit.client.0.trial112.stdout:#---------- 3.0 loop 0 2026-02-13T02:11:41.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-13T02:11:41.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-13T02:11:41.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-13T02:11:41.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-13T02:11:41.706 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-13T02:11:41.706 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-13T02:11:41.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean+inconsistent 2026-02-13T02:11:41.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean+inconsistent == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-13T02:11:41.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-13T02:11:41.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-13T02:11:41.781 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-13T02:11:41.781 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:11:41.782 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:11:41.782 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:11:41.782 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-13T02:11:41.944 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: local last_scrub=2026-02-13T02:10:16.209170+0000 2026-02-13T02:11:41.944 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: pg_deep_scrub: ceph pg deep-scrub 3.0 2026-02-13T02:11:42.092 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2026-02-13T02:11:42.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: pg_deep_scrub: wait_for_scrub 3.0 2026-02-13T02:10:16.209170+0000 last_deep_scrub_stamp 2026-02-13T02:11:42.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-13T02:11:42.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:10:16.209170+0000 2026-02-13T02:11:42.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_deep_scrub_stamp 2026-02-13T02:11:42.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:11:42.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:11:42.103 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-13T02:11:42.103 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:11:42.103 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:11:42.104 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:11:42.104 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-13T02:11:42.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:10:16.209170+0000 '>' 2026-02-13T02:10:16.209170+0000 2026-02-13T02:11:42.269 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:11:43.270 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:11:43.270 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:11:43.271 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-13T02:11:43.271 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:11:43.271 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:11:43.272 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:11:43.272 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-13T02:11:43.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:10:16.209170+0000 '>' 2026-02-13T02:10:16.209170+0000 2026-02-13T02:11:43.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:11:44.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:11:44.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:11:44.438 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-13T02:11:44.438 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:11:44.438 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:11:44.439 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:11:44.439 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-13T02:11:44.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:10:16.209170+0000 '>' 2026-02-13T02:10:16.209170+0000 2026-02-13T02:11:44.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:11:45.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:11:45.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:11:45.604 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-13T02:11:45.605 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:11:45.605 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:11:45.605 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:11:45.605 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-13T02:11:45.768 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:10:16.209170+0000 '>' 2026-02-13T02:10:16.209170+0000 2026-02-13T02:11:45.768 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:11:46.769 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:11:46.769 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:11:46.770 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-13T02:11:46.770 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:11:46.770 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:11:46.771 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:11:46.771 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-13T02:11:46.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:10:16.209170+0000 '>' 2026-02-13T02:10:16.209170+0000 2026-02-13T02:11:46.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:11:47.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:11:47.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:11:47.935 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-13T02:11:47.935 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:11:47.935 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:11:47.936 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:11:47.936 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-13T02:11:48.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:10:16.209170+0000 '>' 2026-02-13T02:10:16.209170+0000 2026-02-13T02:11:48.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:11:49.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:11:49.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:11:49.098 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-13T02:11:49.098 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:11:49.098 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:11:49.098 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:11:49.098 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-13T02:11:49.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:11:42.464039+0000 '>' 2026-02-13T02:10:16.209170+0000 2026-02-13T02:11:49.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:11:49.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4472: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2026-02-13T02:11:49.280 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4474: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2026-02-13T02:11:49.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4474: corrupt_scrub_erasure: test 1 = 1 2026-02-13T02:11:49.293 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4476: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2026-02-13T02:11:49.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4476: corrupt_scrub_erasure: test 3.0 = 3.0 2026-02-13T02:11:49.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4478: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2026-02-13T02:11:49.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4479: corrupt_scrub_erasure: cp td/osd-scrub-repair/json /tmp/rrr6.json 2026-02-13T02:11:49.322 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4481: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2026-02-13T02:11:49.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4481: corrupt_scrub_erasure: epoch=99 2026-02-13T02:11:49.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4483: corrupt_scrub_erasure: '[' false = true ']' 2026-02-13T02:11:49.336 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5148: corrupt_scrub_erasure: jq 'def walk(f): 2026-02-13T02:11:49.336 INFO:tasks.workunit.client.0.trial112.stderr: . as $in 2026-02-13T02:11:49.336 INFO:tasks.workunit.client.0.trial112.stderr: | if type == "object" then 2026-02-13T02:11:49.336 INFO:tasks.workunit.client.0.trial112.stderr: reduce keys[] as $key 2026-02-13T02:11:49.336 INFO:tasks.workunit.client.0.trial112.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-13T02:11:49.337 INFO:tasks.workunit.client.0.trial112.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-13T02:11:49.337 INFO:tasks.workunit.client.0.trial112.stderr: else f 2026-02-13T02:11:49.337 INFO:tasks.workunit.client.0.trial112.stderr: end; 2026-02-13T02:11:49.337 INFO:tasks.workunit.client.0.trial112.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-13T02:11:49.337 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-13T02:11:49.337 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-13T02:11:49.337 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-13T02:11:49.337 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-13T02:11:49.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5148: corrupt_scrub_erasure: jq .inconsistents 2026-02-13T02:11:49.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5148: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-13T02:11:49.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5890: corrupt_scrub_erasure: jq 'def walk(f): 2026-02-13T02:11:49.353 INFO:tasks.workunit.client.0.trial112.stderr: . as $in 2026-02-13T02:11:49.354 INFO:tasks.workunit.client.0.trial112.stderr: | if type == "object" then 2026-02-13T02:11:49.354 INFO:tasks.workunit.client.0.trial112.stderr: reduce keys[] as $key 2026-02-13T02:11:49.354 INFO:tasks.workunit.client.0.trial112.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-13T02:11:49.354 INFO:tasks.workunit.client.0.trial112.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-13T02:11:49.354 INFO:tasks.workunit.client.0.trial112.stderr: else f 2026-02-13T02:11:49.354 INFO:tasks.workunit.client.0.trial112.stderr: end; 2026-02-13T02:11:49.354 INFO:tasks.workunit.client.0.trial112.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-13T02:11:49.354 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-13T02:11:49.354 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-13T02:11:49.354 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-13T02:11:49.355 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2026-02-13T02:11:49.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5890: corrupt_scrub_erasure: jq .inconsistents 2026-02-13T02:11:49.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5890: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-13T02:11:49.372 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5891: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-13T02:11:49.372 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2513: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-13T02:11:49.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5892: corrupt_scrub_erasure: test no = yes 2026-02-13T02:11:49.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5903: corrupt_scrub_erasure: test '' = yes 2026-02-13T02:11:49.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5908: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2026-02-13T02:11:49.585 INFO:tasks.workunit.client.0.trial112.stderr:pool 'ecpool' removed 2026-02-13T02:11:49.598 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-13T02:11:49.599 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:11:49.599 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:11:49.599 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:11:49.599 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:11:49.599 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:11:49.599 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:11:49.599 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:11:49.600 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:11:49.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:11:49.709 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:11:49.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:11:49.711 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:11:49.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:11:49.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:11:49.713 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:11:49.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:11:49.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:11:49.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:11:49.715 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:11:49.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:11:49.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:11:49.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:11:49.743 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:11:49.743 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:11:49.743 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:11:49.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:11:49.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:11:49.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:11:49.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-13T02:11:49.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-13T02:11:49.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-13T02:11:49.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-13T02:11:49.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:11:49.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:11:49.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:11:49.746 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:11:49.746 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:11:49.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:11:49.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:11:49.747 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:11:49.748 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:11:49.748 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:11:49.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:11:49.750 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:11:49.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:11:49.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:11:49.752 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:11:49.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:11:49.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:11:49.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:11:49.753 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:11:49.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:11:49.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:11:49.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:11:49.758 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:11:49.758 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:11:49.758 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:11:49.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:11:49.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:11:49.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:11:49.760 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-13T02:11:49.761 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T02:11:49.761 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:11:49.761 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:11:49.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.35207 2026-02-13T02:11:49.763 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T02:11:49.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-13T02:11:49.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T02:11:49.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-13T02:11:49.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_erasure_overwrites td/osd-scrub-repair 2026-02-13T02:11:49.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5916: TEST_corrupt_scrub_erasure_overwrites: '[' true = true ']' 2026-02-13T02:11:49.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5917: TEST_corrupt_scrub_erasure_overwrites: corrupt_scrub_erasure td/osd-scrub-repair true 2026-02-13T02:11:49.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3736: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2026-02-13T02:11:49.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3737: corrupt_scrub_erasure: local allow_overwrites=true 2026-02-13T02:11:49.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3738: corrupt_scrub_erasure: local poolname=ecpool 2026-02-13T02:11:49.766 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3739: corrupt_scrub_erasure: local total_objs=7 2026-02-13T02:11:49.766 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3741: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2026-02-13T02:11:49.766 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-13T02:11:49.766 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-13T02:11:49.766 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-13T02:11:49.766 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-13T02:11:49.766 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-13T02:11:49.766 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-13T02:11:49.787 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-13T02:11:49.787 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:11:49.787 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:11:49.787 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:11:49.787 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:11:49.787 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:11:49.788 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:11:49.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-13T02:11:49.816 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-13T02:11:49.818 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-13T02:11:49.818 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:11:49.818 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:11:49.818 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-13T02:11:49.819 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-13T02:11:49.819 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:11:49.819 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:11:49.819 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:11:49.819 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:11:49.819 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:11:49.819 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:11:49.820 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:11:49.820 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:11:49.820 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get fsid 2026-02-13T02:11:49.877 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-13T02:11:49.877 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:11:49.877 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:11:49.878 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-13T02:11:49.879 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-13T02:11:49.879 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:11:49.879 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:11:49.879 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:11:49.879 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:11:49.879 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:11:49.879 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:11:49.880 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:11:49.880 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:11:49.880 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get mon_host 2026-02-13T02:11:49.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3742: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2026-02-13T02:11:49.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-13T02:11:49.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-13T02:11:49.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-13T02:11:49.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-13T02:11:49.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-13T02:11:49.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-13T02:11:50.035 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-13T02:11:50.035 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:11:50.035 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:11:50.036 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:11:50.036 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:11:50.036 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:11:50.036 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:11:50.037 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:11:50.037 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:11:50.054 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3743: corrupt_scrub_erasure: seq 0 2 2026-02-13T02:11:50.057 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3743: corrupt_scrub_erasure: for id in $(seq 0 2) 2026-02-13T02:11:50.058 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3744: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 0 2026-02-13T02:11:50.058 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:11:50.058 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:11:50.058 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-13T02:11:50.058 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:11:50.058 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:11:50.058 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:11:50.058 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:11:50.058 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:11:50.058 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:11:50.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:11:50.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:11:50.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:11:50.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:11:50.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:11:50.059 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:11:50.059 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:11:50.059 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:11:50.059 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:11:50.059 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:11:50.059 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:11:50.059 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:11:50.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:11:50.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:11:50.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:11:50.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:11:50.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:11:50.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:11:50.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:11:50.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:11:50.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:11:50.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:11:50.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:11:50.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:11:50.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:11:50.061 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:11:50.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a5bec03f-4e86-4fe9-8351-a3f10d7024e0 2026-02-13T02:11:50.061 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 a5bec03f-4e86-4fe9-8351-a3f10d7024e0 2026-02-13T02:11:50.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 a5bec03f-4e86-4fe9-8351-a3f10d7024e0' 2026-02-13T02:11:50.062 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:11:50.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBmiI5pTLWcAxAAqjQllUUWq58afgE8x/JtGA== 2026-02-13T02:11:50.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBmiI5pTLWcAxAAqjQllUUWq58afgE8x/JtGA=="}' 2026-02-13T02:11:50.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a5bec03f-4e86-4fe9-8351-a3f10d7024e0 -i td/osd-scrub-repair/0/new.json 2026-02-13T02:11:50.161 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:11:50.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-13T02:11:50.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBmiI5pTLWcAxAAqjQllUUWq58afgE8x/JtGA== --osd-uuid a5bec03f-4e86-4fe9-8351-a3f10d7024e0 2026-02-13T02:11:50.186 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:50.174+0000 7f9c6a7f3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:11:50.188 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:50.178+0000 7f9c6a7f3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:11:50.188 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:50.178+0000 7f9c6a7f3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:11:50.189 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:50.178+0000 7f9c6a7f3a40 -1 bdev(0x560204a9ec00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-13T02:11:50.189 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:50.178+0000 7f9c6a7f3a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-13T02:11:50.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-13T02:11:50.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:11:50.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-13T02:11:50.881 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T02:11:50.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:11:50.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-13T02:11:50.986 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T02:11:50.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:11:50.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:11:50.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:11:50.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:11:51.009 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:50.998+0000 7fa0da215a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:11:51.011 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:50.998+0000 7fa0da215a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:11:51.012 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:51.002+0000 7fa0da215a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:11:51.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-13T02:11:51.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:11:51.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:11:51.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:11:51.082 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:11:51.082 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:11:51.082 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:11:51.082 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:11:51.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:11:51.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:11:51.176 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:11:51.489 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:51.478+0000 7fa0da215a40 -1 Falling back to public interface 2026-02-13T02:11:52.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:11:52.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:11:52.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:11:52.178 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:11:52.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:11:52.182 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:11:52.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:11:52.446 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:52.434+0000 7fa0da215a40 -1 osd.0 0 log_to_monitors true 2026-02-13T02:11:53.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:11:53.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:11:53.308 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:11:53.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:11:53.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:11:53.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:11:53.473 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/810778247,v1:127.0.0.1:6801/810778247] [v2:127.0.0.1:6802/810778247,v1:127.0.0.1:6803/810778247] exists,up a5bec03f-4e86-4fe9-8351-a3f10d7024e0 2026-02-13T02:11:53.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:11:53.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:11:53.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:11:53.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3743: corrupt_scrub_erasure: for id in $(seq 0 2) 2026-02-13T02:11:53.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3744: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 1 2026-02-13T02:11:53.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:11:53.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:11:53.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-13T02:11:53.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:11:53.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:11:53.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:11:53.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:11:53.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:11:53.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:11:53.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:11:53.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:11:53.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:11:53.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:11:53.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:11:53.478 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:11:53.478 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:11:53.478 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:11:53.479 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:11:53.479 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:11:53.479 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:11:53.479 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:11:53.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:11:53.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:11:53.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:11:53.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:11:53.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:11:53.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:11:53.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:11:53.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:11:53.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:11:53.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:11:53.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:11:53.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:11:53.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:11:53.480 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:11:53.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4869dc97-b36a-441e-9ba9-92eb0a3698d2 2026-02-13T02:11:53.481 INFO:tasks.workunit.client.0.trial112.stdout:add osd1 4869dc97-b36a-441e-9ba9-92eb0a3698d2 2026-02-13T02:11:53.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 4869dc97-b36a-441e-9ba9-92eb0a3698d2' 2026-02-13T02:11:53.481 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:11:53.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBpiI5pCPO1HBAAyN4aIZ2V4QZHr5muZPsGXw== 2026-02-13T02:11:53.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBpiI5pCPO1HBAAyN4aIZ2V4QZHr5muZPsGXw=="}' 2026-02-13T02:11:53.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4869dc97-b36a-441e-9ba9-92eb0a3698d2 -i td/osd-scrub-repair/1/new.json 2026-02-13T02:11:53.651 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:11:53.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-13T02:11:53.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBpiI5pCPO1HBAAyN4aIZ2V4QZHr5muZPsGXw== --osd-uuid 4869dc97-b36a-441e-9ba9-92eb0a3698d2 2026-02-13T02:11:53.678 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:53.666+0000 7f44af3b6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:11:53.679 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:53.666+0000 7f44af3b6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:11:53.680 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:53.670+0000 7f44af3b6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:11:53.681 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:53.670+0000 7f44af3b6a40 -1 bdev(0x562c86fdac00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-13T02:11:53.681 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:53.670+0000 7f44af3b6a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-13T02:11:54.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-13T02:11:54.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:11:54.351 INFO:tasks.workunit.client.0.trial112.stdout:adding osd1 key to auth repository 2026-02-13T02:11:54.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-13T02:11:54.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:11:54.551 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T02:11:54.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-13T02:11:54.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:11:54.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:11:54.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:11:54.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:11:54.576 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:54.566+0000 7fa2e824ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:11:54.578 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:54.566+0000 7fa2e824ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:11:54.579 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:54.566+0000 7fa2e824ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:11:54.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-13T02:11:54.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:11:54.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:11:54.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:11:54.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:11:54.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:11:54.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:11:54.715 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:11:54.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:11:54.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:11:54.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:11:54.905 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:54.894+0000 7fa2e824ea40 -1 Falling back to public interface 2026-02-13T02:11:55.494 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:55.482+0000 7fa2e824ea40 -1 osd.1 0 log_to_monitors true 2026-02-13T02:11:55.877 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:11:55.877 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:11:55.877 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:11:55.878 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:11:55.878 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:11:55.878 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:11:56.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:11:57.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:11:57.047 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:11:57.047 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:11:57.048 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:11:57.048 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:11:57.048 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:11:57.209 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3442709724,v1:127.0.0.1:6811/3442709724] [v2:127.0.0.1:6812/3442709724,v1:127.0.0.1:6813/3442709724] exists,up 4869dc97-b36a-441e-9ba9-92eb0a3698d2 2026-02-13T02:11:57.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:11:57.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:11:57.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:11:57.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3743: corrupt_scrub_erasure: for id in $(seq 0 2) 2026-02-13T02:11:57.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3744: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 2 2026-02-13T02:11:57.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:11:57.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:11:57.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-13T02:11:57.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:11:57.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-13T02:11:57.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:11:57.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:11:57.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:11:57.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:11:57.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-13T02:11:57.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-13T02:11:57.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:11:57.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:11:57.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:11:57.214 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:11:57.214 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:11:57.214 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:11:57.214 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:11:57.214 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:11:57.214 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:11:57.214 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:11:57.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:11:57.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:11:57.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:11:57.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:11:57.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:11:57.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:11:57.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:11:57.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:11:57.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:11:57.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:11:57.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:11:57.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:11:57.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-13T02:11:57.216 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:11:57.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=06546da0-fa39-4955-a99d-383e02e4b0c5 2026-02-13T02:11:57.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 06546da0-fa39-4955-a99d-383e02e4b0c5' 2026-02-13T02:11:57.216 INFO:tasks.workunit.client.0.trial112.stdout:add osd2 06546da0-fa39-4955-a99d-383e02e4b0c5 2026-02-13T02:11:57.217 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:11:57.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBtiI5pmZDsDBAAKSUulUQT3I30f0eohgK8gQ== 2026-02-13T02:11:57.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBtiI5pmZDsDBAAKSUulUQT3I30f0eohgK8gQ=="}' 2026-02-13T02:11:57.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 06546da0-fa39-4955-a99d-383e02e4b0c5 -i td/osd-scrub-repair/2/new.json 2026-02-13T02:11:57.381 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:11:57.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-13T02:11:57.394 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBtiI5pmZDsDBAAKSUulUQT3I30f0eohgK8gQ== --osd-uuid 06546da0-fa39-4955-a99d-383e02e4b0c5 2026-02-13T02:11:57.418 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:57.406+0000 7f75d7b49a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:11:57.419 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:57.406+0000 7f75d7b49a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:11:57.420 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:57.410+0000 7f75d7b49a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:11:57.421 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:57.410+0000 7f75d7b49a40 -1 bdev(0x55aa9e55ec00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-13T02:11:57.421 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:57.410+0000 7f75d7b49a40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-13T02:11:58.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-13T02:11:58.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:11:58.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-13T02:11:58.021 INFO:tasks.workunit.client.0.trial112.stdout:adding osd2 key to auth repository 2026-02-13T02:11:58.021 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:11:58.232 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-13T02:11:58.233 INFO:tasks.workunit.client.0.trial112.stdout:start osd.2 2026-02-13T02:11:58.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:11:58.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:11:58.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:11:58.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:11:58.257 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:58.246+0000 7efe3b39ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:11:58.258 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:58.246+0000 7efe3b39ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:11:58.259 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:58.246+0000 7efe3b39ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:11:58.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-13T02:11:58.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:11:58.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T02:11:58.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:11:58.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:11:58.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:11:58.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:11:58.388 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:11:58.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:11:58.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:11:58.548 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:11:58.625 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:58.614+0000 7efe3b39ba40 -1 Falling back to public interface 2026-02-13T02:11:59.251 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:11:59.238+0000 7efe3b39ba40 -1 osd.2 0 log_to_monitors true 2026-02-13T02:11:59.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:11:59.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:11:59.549 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:11:59.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:11:59.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:11:59.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:11:59.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:12:00.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:12:00.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:12:00.714 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:12:00.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:12:00.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:12:00.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:12:00.877 INFO:tasks.workunit.client.0.trial112.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3892902953,v1:127.0.0.1:6819/3892902953] [v2:127.0.0.1:6820/3892902953,v1:127.0.0.1:6821/3892902953] exists,up 06546da0-fa39-4955-a99d-383e02e4b0c5 2026-02-13T02:12:00.878 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:12:00.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:12:00.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:12:00.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3746: corrupt_scrub_erasure: create_rbd_pool 2026-02-13T02:12:00.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-13T02:12:01.027 INFO:tasks.workunit.client.0.trial112.stderr:pool 'rbd' does not exist 2026-02-13T02:12:01.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-13T02:12:01.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-13T02:12:01.251 INFO:tasks.workunit.client.0.trial112.stderr:pool 'rbd' created 2026-02-13T02:12:01.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:12:02.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-13T02:12:02.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3747: corrupt_scrub_erasure: create_pool foo 1 2026-02-13T02:12:02.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 1 2026-02-13T02:12:02.741 INFO:tasks.workunit.client.0.trial112.stderr:pool 'foo' created 2026-02-13T02:12:02.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:12:03.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3749: corrupt_scrub_erasure: create_ec_pool ecpool true k=2 m=1 stripe_unit=2K --force 2026-02-13T02:12:03.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2522: create_ec_pool: local pool_name=ecpool 2026-02-13T02:12:03.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2523: create_ec_pool: shift 2026-02-13T02:12:03.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2524: create_ec_pool: local allow_overwrites=true 2026-02-13T02:12:03.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2525: create_ec_pool: shift 2026-02-13T02:12:03.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2527: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2026-02-13T02:12:03.966 INFO:tasks.workunit.client.0.trial112.stderr:load: isa 2026-02-13T02:12:03.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2529: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2026-02-13T02:12:03.980 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2026-02-13T02:12:04.270 INFO:tasks.workunit.client.0.trial112.stderr:pool 'ecpool' created 2026-02-13T02:12:04.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:12:05.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2531: create_ec_pool: '[' true = true ']' 2026-02-13T02:12:05.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2532: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2026-02-13T02:12:05.495 INFO:tasks.workunit.client.0.trial112.stderr:set pool 3 allow_ec_overwrites to true 2026-02-13T02:12:05.507 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2535: create_ec_pool: wait_for_clean 2026-02-13T02:12:05.507 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:12:05.507 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:12:05.507 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:12:05.507 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:12:05.508 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:12:05.508 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:12:05.509 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:12:05.509 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:12:05.509 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:12:05.611 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:12:05.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:12:05.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:12:05.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:12:05.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:12:05.612 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:12:05.771 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:12:05.772 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:12:05.772 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:12:05.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:12:05.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:12:05.772 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:12:05.847 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836484 2026-02-13T02:12:05.847 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836484 2026-02-13T02:12:05.847 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484' 2026-02-13T02:12:05.847 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:12:05.847 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:12:05.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672963 2026-02-13T02:12:05.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672963 2026-02-13T02:12:05.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-42949672963' 2026-02-13T02:12:05.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:12:05.920 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:12:05.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509443 2026-02-13T02:12:05.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509443 2026-02-13T02:12:05.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-42949672963 2-64424509443' 2026-02-13T02:12:05.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:12:05.991 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836484 2026-02-13T02:12:05.991 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:12:05.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:12:05.994 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836484 2026-02-13T02:12:05.994 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:12:05.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836484 2026-02-13T02:12:05.995 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836484 2026-02-13T02:12:05.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-13T02:12:05.996 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:12:06.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836484 2026-02-13T02:12:06.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:12:07.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:12:07.171 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:12:07.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836484 2026-02-13T02:12:07.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:12:07.331 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672963 2026-02-13T02:12:07.331 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:12:07.332 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:12:07.333 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672963 2026-02-13T02:12:07.333 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:12:07.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672963 2026-02-13T02:12:07.335 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 42949672963 2026-02-13T02:12:07.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2026-02-13T02:12:07.336 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:12:07.509 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672963 -lt 42949672963 2026-02-13T02:12:07.509 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:12:07.510 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509443 2026-02-13T02:12:07.510 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:12:07.511 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:12:07.512 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509443 2026-02-13T02:12:07.512 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:12:07.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509443 2026-02-13T02:12:07.514 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 64424509443 2026-02-13T02:12:07.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509443' 2026-02-13T02:12:07.515 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:12:07.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509443 -lt 64424509443 2026-02-13T02:12:07.679 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:12:07.680 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:12:07.680 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:12:07.870 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:12:07.871 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:12:07.871 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:12:07.872 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:12:07.872 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:12:07.872 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:12:07.872 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:12:07.872 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:12:08.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:12:08.027 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:12:08.028 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:12:08.028 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:12:08.218 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:12:08.218 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:12:08.218 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:12:08.218 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2536: create_ec_pool: return 0 2026-02-13T02:12:08.218 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3750: corrupt_scrub_erasure: wait_for_clean 2026-02-13T02:12:08.219 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:12:08.219 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:12:08.219 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:12:08.219 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:12:08.220 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:12:08.220 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:12:08.220 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:12:08.220 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:12:08.220 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:12:08.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:12:08.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:12:08.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:12:08.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:12:08.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:12:08.326 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:12:08.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:12:08.483 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:12:08.483 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:12:08.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:12:08.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:12:08.484 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:12:08.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836485 2026-02-13T02:12:08.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836485 2026-02-13T02:12:08.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485' 2026-02-13T02:12:08.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:12:08.560 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:12:08.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672965 2026-02-13T02:12:08.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672965 2026-02-13T02:12:08.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-42949672965' 2026-02-13T02:12:08.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:12:08.632 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:12:08.704 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509444 2026-02-13T02:12:08.704 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509444 2026-02-13T02:12:08.704 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-42949672965 2-64424509444' 2026-02-13T02:12:08.704 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:12:08.705 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836485 2026-02-13T02:12:08.705 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:12:08.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:12:08.707 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836485 2026-02-13T02:12:08.707 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:12:08.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836485 2026-02-13T02:12:08.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-13T02:12:08.709 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836485 2026-02-13T02:12:08.709 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:12:08.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836485 2026-02-13T02:12:08.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:12:09.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:12:09.875 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:12:10.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836485 2026-02-13T02:12:10.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:12:11.037 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:12:11.037 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:12:11.203 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836485 2026-02-13T02:12:11.203 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:12:11.204 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672965 2026-02-13T02:12:11.204 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:12:11.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:12:11.206 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672965 2026-02-13T02:12:11.206 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:12:11.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672965 2026-02-13T02:12:11.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2026-02-13T02:12:11.208 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 42949672965 2026-02-13T02:12:11.209 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:12:11.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672965 -lt 42949672965 2026-02-13T02:12:11.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:12:11.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509444 2026-02-13T02:12:11.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:12:11.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:12:11.376 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509444 2026-02-13T02:12:11.376 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:12:11.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509444 2026-02-13T02:12:11.378 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 64424509444 2026-02-13T02:12:11.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2026-02-13T02:12:11.378 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:12:11.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509444 -lt 64424509444 2026-02-13T02:12:11.545 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:12:11.546 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:12:11.546 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:12:11.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:12:11.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:12:11.739 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:12:11.739 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:12:11.739 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:12:11.739 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:12:11.739 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:12:11.739 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:12:11.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:12:11.895 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:12:11.896 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:12:11.896 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:12:12.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:12:12.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:12:12.089 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:12:12.089 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: seq 1 7 2026-02-13T02:12:12.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-13T02:12:12.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ1 2026-02-13T02:12:12.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2026-02-13T02:12:12.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:12:12.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-13T02:12:12.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ1 2026-02-13T02:12:12.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:12:12.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:12:12.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:12:12.290 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:12:12.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:12:12.496 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:12:12.507 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:12:12.507 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:12:12.507 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:12:12.529 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 1 % 2 2026-02-13T02:12:12.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=1 2026-02-13T02:12:12.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-13T02:12:12.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3761: corrupt_scrub_erasure: local payload=UVWXYZZZ 2026-02-13T02:12:12.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3762: corrupt_scrub_erasure: echo UVWXYZZZ 2026-02-13T02:12:12.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3763: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-13T02:12:12.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:12:12.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:12:12.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:12:12.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:12:12.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-13T02:12:12.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:12:12.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:12:12.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:12:12.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:12:12.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:12:12.533 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:12:12.533 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:12:12.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:12:12.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:12:12.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:12:12.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:12:12.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-13T02:12:12.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:12:12.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:12:12.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:12:12.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:12:12.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:12:12.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-13T02:12:13.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-13T02:12:13.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:12:13.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:12:13.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:12:13.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:12:13.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:12:13.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:12:13.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:12:13.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:12:13.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:12:13.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:12:13.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:12:13.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:12:13.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:12:13.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:12:13.159 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:12:13.159 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:12:13.159 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:12:13.160 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:12:13.160 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:12:13.160 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:12:13.160 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:12:13.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:12:13.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:12:13.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:12:13.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:12:13.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:12:13.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:12:13.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:12:13.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:12:13.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:12:13.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:12:13.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:12:13.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:12:13.163 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:12:13.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:12:13.163 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:12:13.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:12:13.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:12:13.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:12:13.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:12:13.176 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:13.166+0000 7fc6fc4bea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:12:13.177 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:13.166+0000 7fc6fc4bea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:12:13.178 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:13.166+0000 7fc6fc4bea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:12:13.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:12:13.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:12:13.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:12:13.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:12:13.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:12:13.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:12:13.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:12:13.326 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:12:13.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:12:13.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:12:13.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:12:13.589 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:13.578+0000 7fc6fc4bea40 -1 Falling back to public interface 2026-02-13T02:12:14.188 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:14.178+0000 7fc6fc4bea40 -1 osd.1 34 log_to_monitors true 2026-02-13T02:12:14.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:12:14.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:12:14.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:12:14.489 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:12:14.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:12:14.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:12:14.648 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 38 up_thru 38 down_at 35 last_clean_interval [10,34) [v2:127.0.0.1:6810/2950618490,v1:127.0.0.1:6811/2950618490] [v2:127.0.0.1:6812/2950618490,v1:127.0.0.1:6813/2950618490] exists,up 4869dc97-b36a-441e-9ba9-92eb0a3698d2 2026-02-13T02:12:14.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:12:14.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:12:14.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:12:14.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:12:14.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:12:14.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:12:14.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:12:14.650 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:12:14.650 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:12:14.650 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:12:14.650 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:12:14.650 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:12:14.650 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:12:14.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:12:14.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:12:14.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:12:14.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:12:14.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:12:14.755 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:12:14.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:12:14.923 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:12:14.923 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:12:14.924 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:12:14.924 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:12:14.924 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:12:14.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836488 2026-02-13T02:12:14.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836488 2026-02-13T02:12:14.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488' 2026-02-13T02:12:14.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:12:14.997 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:12:15.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=163208757250 2026-02-13T02:12:15.072 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 163208757250 2026-02-13T02:12:15.072 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-163208757250' 2026-02-13T02:12:15.072 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:12:15.072 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:12:15.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509446 2026-02-13T02:12:15.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509446 2026-02-13T02:12:15.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-163208757250 2-64424509446' 2026-02-13T02:12:15.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:12:15.146 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836488 2026-02-13T02:12:15.146 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:12:15.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:12:15.148 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836488 2026-02-13T02:12:15.149 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:12:15.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836488 2026-02-13T02:12:15.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2026-02-13T02:12:15.150 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 21474836488 2026-02-13T02:12:15.151 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:12:15.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836488 2026-02-13T02:12:15.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:12:16.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:12:16.324 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:12:16.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836488 2026-02-13T02:12:16.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:12:16.483 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-163208757250 2026-02-13T02:12:16.483 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:12:16.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:12:16.485 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-163208757250 2026-02-13T02:12:16.485 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:12:16.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=163208757250 2026-02-13T02:12:16.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 163208757250' 2026-02-13T02:12:16.487 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 163208757250 2026-02-13T02:12:16.487 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:12:16.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 163208757250 -lt 163208757250 2026-02-13T02:12:16.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:12:16.651 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509446 2026-02-13T02:12:16.651 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:12:16.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:12:16.653 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509446 2026-02-13T02:12:16.653 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:12:16.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509446 2026-02-13T02:12:16.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2026-02-13T02:12:16.655 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 64424509446 2026-02-13T02:12:16.655 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:12:16.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509447 -lt 64424509446 2026-02-13T02:12:16.825 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:12:16.826 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:12:16.826 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:12:17.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:12:17.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:12:17.019 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:12:17.019 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:12:17.019 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:12:17.019 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:12:17.020 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:12:17.020 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:12:17.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:12:17.174 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:12:17.175 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:12:17.175 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:12:17.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:12:17.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:12:17.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:12:17.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-13T02:12:17.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ2 2026-02-13T02:12:17.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2026-02-13T02:12:17.368 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:12:17.368 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-13T02:12:17.368 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ2 2026-02-13T02:12:17.368 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:12:17.368 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:12:17.368 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:12:17.572 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:12:17.584 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:12:17.777 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:12:17.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:12:17.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:12:17.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:12:17.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 2 % 2 2026-02-13T02:12:17.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=0 2026-02-13T02:12:17.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-13T02:12:17.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3768: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2026-02-13T02:12:17.814 INFO:tasks.workunit.client.0.trial112.stderr:1+0 records in 2026-02-13T02:12:17.814 INFO:tasks.workunit.client.0.trial112.stderr:1+0 records out 2026-02-13T02:12:17.814 INFO:tasks.workunit.client.0.trial112.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 8.8393e-05 s, 23.2 MB/s 2026-02-13T02:12:17.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3769: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-13T02:12:17.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:12:17.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:12:17.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T02:12:17.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:12:17.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-13T02:12:17.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:12:17.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:12:17.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T02:12:17.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:12:17.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T02:12:17.815 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:12:17.815 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:12:17.816 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:12:17.816 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:12:17.816 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:12:18.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:12:18.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-13T02:12:18.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:12:18.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:12:18.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:12:18.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:12:18.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:12:18.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-13T02:12:18.414 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-13T02:12:18.414 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:12:18.414 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:12:18.414 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:12:18.414 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:12:18.414 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:12:18.414 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:12:18.414 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:12:18.414 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:12:18.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:12:18.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:12:18.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:12:18.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:12:18.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:12:18.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:12:18.415 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:12:18.415 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:12:18.415 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:12:18.415 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:12:18.415 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:12:18.415 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:12:18.417 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:12:18.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:12:18.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:12:18.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:12:18.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:12:18.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:12:18.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:12:18.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:12:18.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:12:18.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:12:18.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:12:18.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:12:18.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:12:18.418 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T02:12:18.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:12:18.418 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T02:12:18.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:12:18.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:12:18.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:12:18.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:12:18.435 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:18.422+0000 7fc23406aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:12:18.437 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:18.426+0000 7fc23406aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:12:18.438 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:18.426+0000 7fc23406aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:12:18.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:12:18.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:12:18.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:12:18.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:12:18.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:12:18.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:12:18.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:12:18.579 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:12:18.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:12:18.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:12:18.748 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:12:18.785 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:18.774+0000 7fc23406aa40 -1 Falling back to public interface 2026-02-13T02:12:19.362 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:19.350+0000 7fc23406aa40 -1 osd.0 41 log_to_monitors true 2026-02-13T02:12:19.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:12:19.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:12:19.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:12:19.750 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:12:19.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:12:19.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:12:19.910 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 45 up_thru 45 down_at 42 last_clean_interval [5,41) [v2:127.0.0.1:6800/1571534597,v1:127.0.0.1:6801/1571534597] [v2:127.0.0.1:6802/1571534597,v1:127.0.0.1:6803/1571534597] exists,up a5bec03f-4e86-4fe9-8351-a3f10d7024e0 2026-02-13T02:12:19.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:12:19.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:12:19.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:12:19.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:12:19.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:12:19.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:12:19.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:12:19.911 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:12:19.912 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:12:19.912 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:12:19.912 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:12:19.913 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:12:19.913 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:12:20.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:12:20.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:12:20.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:12:20.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:12:20.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:12:20.020 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:12:20.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:12:20.179 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:12:20.179 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:12:20.180 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:12:20.180 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:12:20.180 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:12:20.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=193273528322 2026-02-13T02:12:20.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 193273528322 2026-02-13T02:12:20.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-193273528322' 2026-02-13T02:12:20.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:12:20.253 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:12:20.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=163208757252 2026-02-13T02:12:20.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 163208757252 2026-02-13T02:12:20.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-193273528322 1-163208757252' 2026-02-13T02:12:20.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:12:20.329 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:12:20.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509448 2026-02-13T02:12:20.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509448 2026-02-13T02:12:20.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-193273528322 1-163208757252 2-64424509448' 2026-02-13T02:12:20.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:12:20.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-193273528322 2026-02-13T02:12:20.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:12:20.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:12:20.409 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-193273528322 2026-02-13T02:12:20.409 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:12:20.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=193273528322 2026-02-13T02:12:20.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 193273528322' 2026-02-13T02:12:20.411 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 193273528322 2026-02-13T02:12:20.411 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:12:20.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 193273528322 -lt 193273528322 2026-02-13T02:12:20.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:12:20.577 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-163208757252 2026-02-13T02:12:20.578 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:12:20.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:12:20.580 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-163208757252 2026-02-13T02:12:20.580 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:12:20.581 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=163208757252 2026-02-13T02:12:20.581 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 163208757252' 2026-02-13T02:12:20.581 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 163208757252 2026-02-13T02:12:20.582 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:12:20.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 163208757251 -lt 163208757252 2026-02-13T02:12:20.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:12:21.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:12:21.744 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:12:21.908 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 163208757251 -lt 163208757252 2026-02-13T02:12:21.908 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:12:22.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:12:22.910 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:12:23.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 163208757252 -lt 163208757252 2026-02-13T02:12:23.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:12:23.074 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509448 2026-02-13T02:12:23.074 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:12:23.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:12:23.076 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509448 2026-02-13T02:12:23.077 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:12:23.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509448 2026-02-13T02:12:23.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2026-02-13T02:12:23.078 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 64424509448 2026-02-13T02:12:23.078 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:12:23.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509449 -lt 64424509448 2026-02-13T02:12:23.250 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:12:23.251 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:12:23.251 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:12:23.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:12:23.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:12:23.446 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:12:23.446 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:12:23.447 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:12:23.447 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:12:23.447 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:12:23.447 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:12:23.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:12:23.606 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:12:23.607 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:12:23.607 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:12:23.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:12:23.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:12:23.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:12:23.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-13T02:12:23.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ3 2026-02-13T02:12:23.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2026-02-13T02:12:23.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:12:23.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-13T02:12:23.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ3 2026-02-13T02:12:23.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:12:23.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:12:23.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:12:24.010 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:12:24.023 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:12:24.214 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:12:24.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:12:24.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:12:24.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:12:24.248 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 3 % 2 2026-02-13T02:12:24.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=1 2026-02-13T02:12:24.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-13T02:12:24.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3774: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2026-02-13T02:12:24.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:12:24.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:12:24.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:12:24.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:12:24.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2026-02-13T02:12:24.251 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:12:24.251 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:12:24.251 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:12:24.251 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:12:24.251 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:12:24.251 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:12:24.251 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:12:24.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:12:24.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:12:24.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:12:24.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:12:24.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2026-02-13T02:12:24.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:12:24.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:12:24.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:12:24.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:12:24.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:12:24.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2026-02-13T02:12:24.735 INFO:tasks.workunit.client.0.trial112.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2026-02-13T02:12:24.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-13T02:12:24.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:12:24.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:12:24.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:12:24.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:12:24.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:12:24.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:12:24.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:12:24.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:12:24.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:12:24.827 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:12:24.827 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:12:24.827 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:12:24.827 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:12:24.827 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:12:24.827 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:12:24.827 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:12:24.827 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:12:24.827 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:12:24.827 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:12:24.827 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:12:24.829 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:12:24.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:12:24.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:12:24.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:12:24.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:12:24.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:12:24.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:12:24.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:12:24.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:12:24.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:12:24.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:12:24.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:12:24.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:12:24.830 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:12:24.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:12:24.830 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:12:24.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:12:24.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:12:24.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:12:24.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:12:24.844 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:24.833+0000 7f741d473a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:12:24.845 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:24.833+0000 7f741d473a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:12:24.846 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:24.833+0000 7f741d473a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:12:24.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:12:24.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:12:24.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:12:24.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:12:24.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:12:24.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:12:24.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:12:24.993 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:12:24.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:12:24.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:12:25.152 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:12:25.193 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:25.181+0000 7f741d473a40 -1 Falling back to public interface 2026-02-13T02:12:25.763 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:25.749+0000 7f741d473a40 -1 osd.1 49 log_to_monitors true 2026-02-13T02:12:26.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:12:26.154 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:12:26.154 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:12:26.154 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:12:26.154 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:12:26.154 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:12:26.310 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 53 up_thru 53 down_at 50 last_clean_interval [38,49) [v2:127.0.0.1:6810/1418799901,v1:127.0.0.1:6811/1418799901] [v2:127.0.0.1:6812/1418799901,v1:127.0.0.1:6813/1418799901] exists,up 4869dc97-b36a-441e-9ba9-92eb0a3698d2 2026-02-13T02:12:26.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:12:26.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:12:26.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:12:26.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:12:26.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:12:26.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:12:26.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:12:26.311 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:12:26.312 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:12:26.312 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:12:26.312 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:12:26.313 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:12:26.313 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:12:26.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:12:26.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:12:26.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:12:26.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:12:26.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:12:26.429 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:12:26.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:12:26.590 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:12:26.591 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:12:26.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:12:26.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:12:26.591 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:12:26.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=193273528324 2026-02-13T02:12:26.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 193273528324 2026-02-13T02:12:26.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-193273528324' 2026-02-13T02:12:26.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:12:26.664 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:12:26.739 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=227633266690 2026-02-13T02:12:26.739 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 227633266690 2026-02-13T02:12:26.739 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-193273528324 1-227633266690' 2026-02-13T02:12:26.739 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:12:26.739 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:12:26.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509451 2026-02-13T02:12:26.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509451 2026-02-13T02:12:26.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-193273528324 1-227633266690 2-64424509451' 2026-02-13T02:12:26.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:12:26.813 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-193273528324 2026-02-13T02:12:26.813 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:12:26.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:12:26.815 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-193273528324 2026-02-13T02:12:26.815 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:12:26.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=193273528324 2026-02-13T02:12:26.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 193273528324' 2026-02-13T02:12:26.817 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 193273528324 2026-02-13T02:12:26.817 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:12:26.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 193273528323 -lt 193273528324 2026-02-13T02:12:26.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:12:27.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:12:27.977 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:12:28.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 193273528323 -lt 193273528324 2026-02-13T02:12:28.144 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:12:29.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:12:29.145 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:12:29.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 193273528324 -lt 193273528324 2026-02-13T02:12:29.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:12:29.310 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-227633266690 2026-02-13T02:12:29.311 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:12:29.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:12:29.313 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-227633266690 2026-02-13T02:12:29.313 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:12:29.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=227633266690 2026-02-13T02:12:29.315 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 227633266690' 2026-02-13T02:12:29.315 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 227633266690 2026-02-13T02:12:29.315 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:12:29.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 227633266690 -lt 227633266690 2026-02-13T02:12:29.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:12:29.484 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509451 2026-02-13T02:12:29.484 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:12:29.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:12:29.485 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509451 2026-02-13T02:12:29.486 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:12:29.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509451 2026-02-13T02:12:29.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2026-02-13T02:12:29.487 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 64424509451 2026-02-13T02:12:29.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:12:29.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509451 -lt 64424509451 2026-02-13T02:12:29.652 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:12:29.652 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:12:29.653 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:12:29.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:12:29.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:12:29.850 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:12:29.850 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:12:29.851 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:12:29.851 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:12:29.851 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:12:29.851 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:12:30.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:12:30.014 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:12:30.014 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:12:30.014 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:12:30.203 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:12:30.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:12:30.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:12:30.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-13T02:12:30.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ4 2026-02-13T02:12:30.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2026-02-13T02:12:30.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:12:30.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-13T02:12:30.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ4 2026-02-13T02:12:30.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:12:30.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:12:30.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:12:30.413 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:12:30.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:12:30.615 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:12:30.625 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:12:30.625 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:12:30.625 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:12:30.646 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 4 % 2 2026-02-13T02:12:30.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=0 2026-02-13T02:12:30.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-13T02:12:30.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3778: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2026-02-13T02:12:30.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3779: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2026-02-13T02:12:30.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3782: corrupt_scrub_erasure: echo -n bad-val 2026-02-13T02:12:30.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3783: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2026-02-13T02:12:30.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:12:30.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:12:30.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T02:12:30.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:12:30.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2026-02-13T02:12:30.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:12:30.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:12:30.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T02:12:30.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:12:30.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T02:12:30.698 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:12:30.698 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:12:30.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:12:30.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:12:30.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:12:31.006 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:12:31.006 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2026-02-13T02:12:31.006 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:12:31.006 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:12:31.006 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:12:31.006 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:12:31.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:12:31.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2026-02-13T02:12:31.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-13T02:12:31.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:12:31.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:12:31.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:12:31.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:12:31.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:12:31.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:12:31.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:12:31.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:12:31.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:12:31.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:12:31.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:12:31.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:12:31.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:12:31.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:12:31.279 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:12:31.279 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:12:31.280 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:12:31.280 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:12:31.280 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:12:31.280 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:12:31.280 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:12:31.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:12:31.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:12:31.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:12:31.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:12:31.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:12:31.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:12:31.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:12:31.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:12:31.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:12:31.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:12:31.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:12:31.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:12:31.283 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T02:12:31.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:12:31.283 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T02:12:31.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:12:31.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:12:31.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:12:31.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:12:31.297 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:31.285+0000 7f13828d2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:12:31.298 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:31.285+0000 7f13828d2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:12:31.299 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:31.285+0000 7f13828d2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:12:31.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:12:31.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:12:31.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:12:31.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:12:31.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:12:31.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:12:31.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:12:31.444 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:12:31.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:12:31.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:12:31.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:12:31.706 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:31.693+0000 7f13828d2a40 -1 Falling back to public interface 2026-02-13T02:12:32.320 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:32.305+0000 7f13828d2a40 -1 osd.0 56 log_to_monitors true 2026-02-13T02:12:32.611 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:12:32.611 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:12:32.611 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:12:32.611 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:12:32.611 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:12:32.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:12:32.771 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 60 up_thru 60 down_at 57 last_clean_interval [45,56) [v2:127.0.0.1:6800/3632719432,v1:127.0.0.1:6801/3632719432] [v2:127.0.0.1:6802/3632719432,v1:127.0.0.1:6803/3632719432] exists,up a5bec03f-4e86-4fe9-8351-a3f10d7024e0 2026-02-13T02:12:32.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:12:32.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:12:32.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:12:32.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:12:32.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:12:32.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:12:32.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:12:32.772 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:12:32.773 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:12:32.773 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:12:32.774 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:12:32.774 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:12:32.774 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:12:32.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:12:32.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:12:32.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:12:32.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:12:32.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:12:32.890 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:12:33.054 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:12:33.054 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:12:33.054 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:12:33.054 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:12:33.054 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:12:33.055 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:12:33.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=257698037762 2026-02-13T02:12:33.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 257698037762 2026-02-13T02:12:33.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-257698037762' 2026-02-13T02:12:33.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:12:33.127 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:12:33.199 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=227633266692 2026-02-13T02:12:33.199 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 227633266692 2026-02-13T02:12:33.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-257698037762 1-227633266692' 2026-02-13T02:12:33.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:12:33.200 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:12:33.270 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509453 2026-02-13T02:12:33.271 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509453 2026-02-13T02:12:33.271 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-257698037762 1-227633266692 2-64424509453' 2026-02-13T02:12:33.271 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:12:33.272 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-257698037762 2026-02-13T02:12:33.272 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:12:33.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:12:33.273 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-257698037762 2026-02-13T02:12:33.274 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:12:33.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=257698037762 2026-02-13T02:12:33.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 257698037762' 2026-02-13T02:12:33.275 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 257698037762 2026-02-13T02:12:33.276 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:12:33.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 193273528325 -lt 257698037762 2026-02-13T02:12:33.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:12:34.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:12:34.447 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:12:34.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 257698037762 -lt 257698037762 2026-02-13T02:12:34.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:12:34.606 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-227633266692 2026-02-13T02:12:34.606 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:12:34.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:12:34.608 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-227633266692 2026-02-13T02:12:34.608 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:12:34.610 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=227633266692 2026-02-13T02:12:34.610 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 227633266692' 2026-02-13T02:12:34.610 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 227633266692 2026-02-13T02:12:34.610 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:12:34.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 227633266692 -lt 227633266692 2026-02-13T02:12:34.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:12:34.786 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509453 2026-02-13T02:12:34.786 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:12:34.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:12:34.788 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509453 2026-02-13T02:12:34.788 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:12:34.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509453 2026-02-13T02:12:34.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2026-02-13T02:12:34.790 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 64424509453 2026-02-13T02:12:34.791 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:12:34.959 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509453 -lt 64424509453 2026-02-13T02:12:34.959 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:12:34.960 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:12:34.960 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:12:35.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:12:35.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:12:35.156 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:12:35.156 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:12:35.156 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:12:35.157 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:12:35.157 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:12:35.157 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:12:35.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:12:35.313 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:12:35.314 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:12:35.314 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:12:35.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:12:35.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:12:35.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:12:35.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3784: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2026-02-13T02:12:35.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:12:35.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:12:35.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-13T02:12:35.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:12:35.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2026-02-13T02:12:35.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:12:35.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:12:35.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-13T02:12:35.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:12:35.505 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2026-02-13T02:12:35.505 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:12:35.505 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:12:35.505 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:12:35.505 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:12:35.505 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:12:35.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:12:35.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2026-02-13T02:12:35.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:12:35.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:12:35.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-13T02:12:35.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:12:35.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2026-02-13T02:12:35.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2026-02-13T02:12:36.082 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2026-02-13T02:12:36.082 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:12:36.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:12:36.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-13T02:12:36.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:12:36.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2026-02-13T02:12:36.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:12:36.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:12:36.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:12:36.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:12:36.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-13T02:12:36.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-13T02:12:36.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:12:36.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:12:36.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:12:36.084 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:12:36.084 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:12:36.084 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:12:36.084 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:12:36.084 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:12:36.084 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:12:36.084 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:12:36.086 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:12:36.086 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:12:36.086 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:12:36.086 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:12:36.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:12:36.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:12:36.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:12:36.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:12:36.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:12:36.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:12:36.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2026-02-13T02:12:36.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-13T02:12:36.087 INFO:tasks.workunit.client.0.trial112.stderr:start osd.2 2026-02-13T02:12:36.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:12:36.087 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2026-02-13T02:12:36.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-13T02:12:36.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:12:36.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:12:36.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:12:36.101 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:36.089+0000 7fb3a3a43a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:12:36.102 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:36.089+0000 7fb3a3a43a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:12:36.104 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:36.089+0000 7fb3a3a43a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:12:36.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-13T02:12:36.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:12:36.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T02:12:36.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:12:36.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:12:36.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:12:36.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:12:36.246 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:12:36.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:12:36.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:12:36.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:12:36.478 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:36.465+0000 7fb3a3a43a40 -1 Falling back to public interface 2026-02-13T02:12:37.047 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:37.033+0000 7fb3a3a43a40 -1 osd.2 63 log_to_monitors true 2026-02-13T02:12:37.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:12:37.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:12:37.416 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:12:37.416 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:12:37.416 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:12:37.416 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:12:37.575 INFO:tasks.workunit.client.0.trial112.stderr:osd.2 up in weight 1 up_from 66 up_thru 66 down_at 63 last_clean_interval [15,63) [v2:127.0.0.1:6818/90985293,v1:127.0.0.1:6819/90985293] [v2:127.0.0.1:6820/90985293,v1:127.0.0.1:6821/90985293] exists,up 06546da0-fa39-4955-a99d-383e02e4b0c5 2026-02-13T02:12:37.576 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:12:37.576 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:12:37.576 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:12:37.576 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:12:37.576 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:12:37.576 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:12:37.576 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:12:37.577 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:12:37.577 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:12:37.577 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:12:37.578 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:12:37.578 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:12:37.578 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:12:37.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:12:37.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:12:37.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:12:37.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:12:37.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:12:37.690 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:12:37.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:12:37.861 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:12:37.861 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:12:37.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:12:37.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:12:37.862 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:12:37.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=257698037764 2026-02-13T02:12:37.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 257698037764 2026-02-13T02:12:37.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-257698037764' 2026-02-13T02:12:37.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:12:37.936 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:12:38.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=227633266694 2026-02-13T02:12:38.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 227633266694 2026-02-13T02:12:38.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-257698037764 1-227633266694' 2026-02-13T02:12:38.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:12:38.008 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:12:38.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=283467841538 2026-02-13T02:12:38.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 283467841538 2026-02-13T02:12:38.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-257698037764 1-227633266694 2-283467841538' 2026-02-13T02:12:38.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:12:38.081 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-257698037764 2026-02-13T02:12:38.081 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:12:38.082 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:12:38.083 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-257698037764 2026-02-13T02:12:38.083 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:12:38.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=257698037764 2026-02-13T02:12:38.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 257698037764' 2026-02-13T02:12:38.085 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 257698037764 2026-02-13T02:12:38.085 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:12:38.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 257698037762 -lt 257698037764 2026-02-13T02:12:38.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:12:39.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:12:39.249 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:12:39.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 257698037764 -lt 257698037764 2026-02-13T02:12:39.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:12:39.409 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-227633266694 2026-02-13T02:12:39.410 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:12:39.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:12:39.412 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-227633266694 2026-02-13T02:12:39.412 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:12:39.414 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=227633266694 2026-02-13T02:12:39.414 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 227633266694' 2026-02-13T02:12:39.414 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 227633266694 2026-02-13T02:12:39.414 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:12:39.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 227633266694 -lt 227633266694 2026-02-13T02:12:39.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:12:39.589 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-283467841538 2026-02-13T02:12:39.589 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:12:39.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:12:39.591 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-283467841538 2026-02-13T02:12:39.591 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:12:39.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=283467841538 2026-02-13T02:12:39.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 283467841538' 2026-02-13T02:12:39.593 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 283467841538 2026-02-13T02:12:39.594 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:12:39.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 283467841538 -lt 283467841538 2026-02-13T02:12:39.753 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:12:39.753 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:12:39.754 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:12:39.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:12:39.944 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:12:39.944 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:12:39.944 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:12:39.944 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:12:39.944 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:12:39.944 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:12:39.944 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:12:40.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:12:40.101 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:12:40.102 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:12:40.102 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:12:40.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:12:40.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:12:40.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:12:40.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3785: corrupt_scrub_erasure: echo -n val3-EOBJ4 2026-02-13T02:12:40.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3786: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2026-02-13T02:12:40.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:12:40.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:12:40.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-13T02:12:40.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:12:40.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2026-02-13T02:12:40.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:12:40.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:12:40.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-13T02:12:40.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:12:40.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2026-02-13T02:12:40.297 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:12:40.297 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:12:40.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:12:40.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:12:40.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:12:40.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:12:40.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2026-02-13T02:12:40.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:12:40.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:12:40.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-13T02:12:40.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:12:40.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2026-02-13T02:12:40.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2026-02-13T02:12:40.872 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2026-02-13T02:12:40.872 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:12:40.872 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:12:40.872 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-13T02:12:40.872 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:12:40.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2026-02-13T02:12:40.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:12:40.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:12:40.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:12:40.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:12:40.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-13T02:12:40.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-13T02:12:40.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:12:40.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:12:40.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:12:40.874 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:12:40.874 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:12:40.874 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:12:40.874 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:12:40.874 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:12:40.874 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:12:40.874 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:12:40.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:12:40.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:12:40.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:12:40.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:12:40.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:12:40.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:12:40.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:12:40.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:12:40.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:12:40.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:12:40.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2026-02-13T02:12:40.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-13T02:12:40.875 INFO:tasks.workunit.client.0.trial112.stderr:start osd.2 2026-02-13T02:12:40.876 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:12:40.877 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2026-02-13T02:12:40.877 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-13T02:12:40.877 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:12:40.877 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:12:40.877 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:12:40.891 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:40.877+0000 7fc7eb467a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:12:40.893 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:40.881+0000 7fc7eb467a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:12:40.894 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:40.881+0000 7fc7eb467a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:12:41.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-13T02:12:41.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:12:41.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T02:12:41.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:12:41.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:12:41.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:12:41.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:12:41.033 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:12:41.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:12:41.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:12:41.201 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:12:41.314 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:41.301+0000 7fc7eb467a40 -1 Falling back to public interface 2026-02-13T02:12:41.932 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:41.917+0000 7fc7eb467a40 -1 osd.2 67 log_to_monitors true 2026-02-13T02:12:42.203 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:12:42.203 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:12:42.203 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:12:42.203 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:12:42.203 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:12:42.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:12:42.364 INFO:tasks.workunit.client.0.trial112.stderr:osd.2 up in weight 1 up_from 71 up_thru 71 down_at 68 last_clean_interval [66,67) [v2:127.0.0.1:6818/2344728558,v1:127.0.0.1:6819/2344728558] [v2:127.0.0.1:6820/2344728558,v1:127.0.0.1:6821/2344728558] exists,up 06546da0-fa39-4955-a99d-383e02e4b0c5 2026-02-13T02:12:42.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:12:42.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:12:42.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:12:42.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:12:42.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:12:42.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:12:42.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:12:42.365 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:12:42.366 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:12:42.366 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:12:42.367 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:12:42.367 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:12:42.367 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:12:42.469 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:12:42.469 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:12:42.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:12:42.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:12:42.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:12:42.470 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:12:42.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:12:42.630 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:12:42.630 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:12:42.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:12:42.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:12:42.630 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:12:42.704 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=257698037766 2026-02-13T02:12:42.704 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 257698037766 2026-02-13T02:12:42.705 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-257698037766' 2026-02-13T02:12:42.705 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:12:42.705 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:12:42.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=227633266696 2026-02-13T02:12:42.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 227633266696 2026-02-13T02:12:42.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-257698037766 1-227633266696' 2026-02-13T02:12:42.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:12:42.776 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:12:42.848 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=304942678018 2026-02-13T02:12:42.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 304942678018 2026-02-13T02:12:42.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-257698037766 1-227633266696 2-304942678018' 2026-02-13T02:12:42.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:12:42.850 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-257698037766 2026-02-13T02:12:42.850 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:12:42.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:12:42.852 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-257698037766 2026-02-13T02:12:42.852 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:12:42.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=257698037766 2026-02-13T02:12:42.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 257698037766' 2026-02-13T02:12:42.854 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 257698037766 2026-02-13T02:12:42.854 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:12:43.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 257698037764 -lt 257698037766 2026-02-13T02:12:43.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:12:44.015 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:12:44.016 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:12:44.176 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 257698037764 -lt 257698037766 2026-02-13T02:12:44.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:12:45.178 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:12:45.178 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:12:45.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 257698037766 -lt 257698037766 2026-02-13T02:12:45.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:12:45.339 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-227633266696 2026-02-13T02:12:45.339 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:12:45.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:12:45.341 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-227633266696 2026-02-13T02:12:45.341 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:12:45.343 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=227633266696 2026-02-13T02:12:45.343 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 227633266696' 2026-02-13T02:12:45.343 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 227633266696 2026-02-13T02:12:45.344 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:12:45.513 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 227633266696 -lt 227633266696 2026-02-13T02:12:45.513 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:12:45.514 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-304942678018 2026-02-13T02:12:45.514 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:12:45.515 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:12:45.516 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-304942678018 2026-02-13T02:12:45.516 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:12:45.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=304942678018 2026-02-13T02:12:45.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 304942678018' 2026-02-13T02:12:45.518 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 304942678018 2026-02-13T02:12:45.518 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:12:45.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 304942678018 -lt 304942678018 2026-02-13T02:12:45.691 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:12:45.691 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:12:45.691 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:12:45.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:12:45.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:12:45.886 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:12:45.886 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:12:45.887 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:12:45.887 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:12:45.887 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:12:45.887 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:12:46.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:12:46.046 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:12:46.046 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:12:46.046 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:12:46.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:12:46.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:12:46.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:12:46.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3787: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2026-02-13T02:12:46.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-13T02:12:46.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ5 2026-02-13T02:12:46.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2026-02-13T02:12:46.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:12:46.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-13T02:12:46.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ5 2026-02-13T02:12:46.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:12:46.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:12:46.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:12:46.478 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:12:46.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:12:46.681 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:12:46.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:12:46.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:12:46.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:12:46.712 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 5 % 2 2026-02-13T02:12:46.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=1 2026-02-13T02:12:46.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-13T02:12:46.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3792: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2026-02-13T02:12:46.716 INFO:tasks.workunit.client.0.trial112.stderr:2+0 records in 2026-02-13T02:12:46.716 INFO:tasks.workunit.client.0.trial112.stderr:2+0 records out 2026-02-13T02:12:46.716 INFO:tasks.workunit.client.0.trial112.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 9.8181e-05 s, 41.7 MB/s 2026-02-13T02:12:46.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3793: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-13T02:12:46.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:12:46.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:12:46.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:12:46.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:12:46.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-13T02:12:46.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:12:46.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:12:46.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:12:46.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:12:46.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:12:46.717 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:12:46.717 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:12:46.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:12:46.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:12:46.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:12:47.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:12:47.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-13T02:12:47.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:12:47.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:12:47.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:12:47.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:12:47.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:12:47.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-13T02:12:47.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-13T02:12:47.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:12:47.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:12:47.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:12:47.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:12:47.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:12:47.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:12:47.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:12:47.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:12:47.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:12:47.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:12:47.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:12:47.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:12:47.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:12:47.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:12:47.305 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:12:47.305 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:12:47.305 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:12:47.305 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:12:47.306 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:12:47.306 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:12:47.308 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:12:47.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:12:47.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:12:47.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:12:47.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:12:47.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:12:47.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:12:47.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:12:47.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:12:47.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:12:47.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:12:47.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:12:47.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:12:47.309 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:12:47.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:12:47.309 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:12:47.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:12:47.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:12:47.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:12:47.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:12:47.329 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:47.317+0000 7fbd6fcb5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:12:47.330 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:47.317+0000 7fbd6fcb5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:12:47.331 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:47.317+0000 7fbd6fcb5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:12:47.465 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:12:47.465 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:12:47.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:12:47.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:12:47.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:12:47.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:12:47.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:12:47.466 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:12:47.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:12:47.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:12:47.626 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:12:47.734 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:47.721+0000 7fbd6fcb5a40 -1 Falling back to public interface 2026-02-13T02:12:48.309 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:48.297+0000 7fbd6fcb5a40 -1 osd.1 74 log_to_monitors true 2026-02-13T02:12:48.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:12:48.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:12:48.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:12:48.629 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:12:48.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:12:48.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:12:48.788 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 78 up_thru 78 down_at 75 last_clean_interval [53,74) [v2:127.0.0.1:6810/3615075866,v1:127.0.0.1:6811/3615075866] [v2:127.0.0.1:6812/3615075866,v1:127.0.0.1:6813/3615075866] exists,up 4869dc97-b36a-441e-9ba9-92eb0a3698d2 2026-02-13T02:12:48.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:12:48.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:12:48.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:12:48.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:12:48.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:12:48.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:12:48.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:12:48.789 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:12:48.790 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:12:48.790 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:12:48.790 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:12:48.790 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:12:48.790 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:12:48.899 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:12:48.899 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:12:48.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:12:48.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:12:48.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:12:48.900 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:12:49.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:12:49.059 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:12:49.059 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:12:49.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:12:49.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:12:49.060 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:12:49.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=257698037768 2026-02-13T02:12:49.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 257698037768 2026-02-13T02:12:49.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-257698037768' 2026-02-13T02:12:49.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:12:49.135 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:12:49.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=335007449090 2026-02-13T02:12:49.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 335007449090 2026-02-13T02:12:49.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-257698037768 1-335007449090' 2026-02-13T02:12:49.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:12:49.209 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:12:49.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=304942678020 2026-02-13T02:12:49.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 304942678020 2026-02-13T02:12:49.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-257698037768 1-335007449090 2-304942678020' 2026-02-13T02:12:49.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:12:49.283 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-257698037768 2026-02-13T02:12:49.284 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:12:49.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:12:49.286 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-257698037768 2026-02-13T02:12:49.286 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:12:49.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=257698037768 2026-02-13T02:12:49.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 257698037768' 2026-02-13T02:12:49.288 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 257698037768 2026-02-13T02:12:49.288 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:12:49.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 257698037767 -lt 257698037768 2026-02-13T02:12:49.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:12:50.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:12:50.463 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:12:50.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 257698037768 -lt 257698037768 2026-02-13T02:12:50.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:12:50.630 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-335007449090 2026-02-13T02:12:50.630 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:12:50.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:12:50.632 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-335007449090 2026-02-13T02:12:50.632 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:12:50.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=335007449090 2026-02-13T02:12:50.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 335007449090' 2026-02-13T02:12:50.634 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 335007449090 2026-02-13T02:12:50.634 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:12:50.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 335007449090 -lt 335007449090 2026-02-13T02:12:50.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:12:50.805 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-304942678020 2026-02-13T02:12:50.805 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:12:50.806 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:12:50.807 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-304942678020 2026-02-13T02:12:50.807 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:12:50.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=304942678020 2026-02-13T02:12:50.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 304942678020' 2026-02-13T02:12:50.809 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 304942678020 2026-02-13T02:12:50.809 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:12:50.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 304942678020 -lt 304942678020 2026-02-13T02:12:50.975 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:12:50.975 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:12:50.975 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:12:51.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:12:51.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:12:51.173 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:12:51.173 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:12:51.173 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:12:51.173 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:12:51.174 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:12:51.174 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:12:51.336 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:12:51.337 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:12:51.337 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:12:51.337 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:12:51.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:12:51.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:12:51.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:12:51.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-13T02:12:51.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ6 2026-02-13T02:12:51.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2026-02-13T02:12:51.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:12:51.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-13T02:12:51.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ6 2026-02-13T02:12:51.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:12:51.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:12:51.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:12:51.736 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:12:51.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:12:51.940 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:12:51.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:12:51.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:12:51.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:12:51.974 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 6 % 2 2026-02-13T02:12:51.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=0 2026-02-13T02:12:51.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-13T02:12:51.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3797: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2026-02-13T02:12:51.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:12:51.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:12:51.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T02:12:51.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:12:51.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2026-02-13T02:12:51.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:12:51.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:12:51.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T02:12:51.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:12:51.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T02:12:51.978 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:12:51.978 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:12:51.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:12:51.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:12:51.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:12:52.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:12:52.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2026-02-13T02:12:52.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:12:52.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:12:52.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:12:52.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:12:52.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:12:52.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2026-02-13T02:12:52.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-13T02:12:52.568 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:12:52.568 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:12:52.568 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:12:52.568 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:12:52.568 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:12:52.568 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:12:52.568 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:12:52.568 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:12:52.568 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:12:52.568 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:12:52.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:12:52.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:12:52.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:12:52.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:12:52.569 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:12:52.569 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:12:52.569 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:12:52.569 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:12:52.569 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:12:52.569 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:12:52.569 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:12:52.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:12:52.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:12:52.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:12:52.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:12:52.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:12:52.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:12:52.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:12:52.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:12:52.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:12:52.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:12:52.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:12:52.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:12:52.572 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T02:12:52.572 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:12:52.572 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T02:12:52.572 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:12:52.572 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:12:52.572 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:12:52.572 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:12:52.588 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:52.573+0000 7fa84b36fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:12:52.589 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:52.577+0000 7fa84b36fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:12:52.590 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:52.577+0000 7fa84b36fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:12:52.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:12:52.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:12:52.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:12:52.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:12:52.737 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:12:52.737 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:12:52.737 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:12:52.737 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:12:52.737 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:12:52.737 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:12:52.898 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:12:53.022 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:53.009+0000 7fa84b36fa40 -1 Falling back to public interface 2026-02-13T02:12:53.623 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:53.609+0000 7fa84b36fa40 -1 osd.0 81 log_to_monitors true 2026-02-13T02:12:53.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:12:53.901 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:12:53.901 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:12:53.901 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:12:53.901 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:12:53.901 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:12:54.060 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 85 up_thru 85 down_at 82 last_clean_interval [60,81) [v2:127.0.0.1:6800/1688195849,v1:127.0.0.1:6801/1688195849] [v2:127.0.0.1:6802/1688195849,v1:127.0.0.1:6803/1688195849] exists,up a5bec03f-4e86-4fe9-8351-a3f10d7024e0 2026-02-13T02:12:54.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:12:54.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:12:54.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:12:54.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:12:54.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:12:54.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:12:54.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:12:54.061 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:12:54.062 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:12:54.062 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:12:54.062 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:12:54.062 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:12:54.062 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:12:54.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:12:54.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:12:54.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:12:54.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:12:54.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:12:54.174 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:12:54.343 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:12:54.343 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:12:54.343 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:12:54.343 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:12:54.343 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:12:54.343 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:12:54.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=365072220162 2026-02-13T02:12:54.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 365072220162 2026-02-13T02:12:54.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-365072220162' 2026-02-13T02:12:54.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:12:54.416 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:12:54.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=335007449092 2026-02-13T02:12:54.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 335007449092 2026-02-13T02:12:54.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-365072220162 1-335007449092' 2026-02-13T02:12:54.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:12:54.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:12:54.572 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=304942678022 2026-02-13T02:12:54.572 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 304942678022 2026-02-13T02:12:54.572 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-365072220162 1-335007449092 2-304942678022' 2026-02-13T02:12:54.572 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:12:54.573 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-365072220162 2026-02-13T02:12:54.573 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:12:54.575 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:12:54.575 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-365072220162 2026-02-13T02:12:54.576 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:12:54.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=365072220162 2026-02-13T02:12:54.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 365072220162' 2026-02-13T02:12:54.577 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 365072220162 2026-02-13T02:12:54.578 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:12:54.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 365072220162 2026-02-13T02:12:54.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:12:55.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:12:55.742 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:12:55.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 365072220162 2026-02-13T02:12:55.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:12:56.904 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:12:56.905 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:12:57.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 365072220162 -lt 365072220162 2026-02-13T02:12:57.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:12:57.069 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-335007449092 2026-02-13T02:12:57.069 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:12:57.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:12:57.071 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-335007449092 2026-02-13T02:12:57.072 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:12:57.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=335007449092 2026-02-13T02:12:57.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 335007449092' 2026-02-13T02:12:57.073 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 335007449092 2026-02-13T02:12:57.074 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:12:57.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 335007449092 -lt 335007449092 2026-02-13T02:12:57.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:12:57.236 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-304942678022 2026-02-13T02:12:57.236 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:12:57.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:12:57.238 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-304942678022 2026-02-13T02:12:57.238 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:12:57.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=304942678022 2026-02-13T02:12:57.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 304942678022' 2026-02-13T02:12:57.240 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 304942678022 2026-02-13T02:12:57.240 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:12:57.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 304942678022 -lt 304942678022 2026-02-13T02:12:57.412 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:12:57.413 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:12:57.413 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:12:57.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:12:57.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:12:57.605 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:12:57.605 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:12:57.605 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:12:57.605 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:12:57.605 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:12:57.605 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:12:57.760 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:12:57.760 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:12:57.760 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:12:57.761 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:12:57.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:12:57.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:12:57.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:12:57.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3798: corrupt_scrub_erasure: echo -n bad-val 2026-02-13T02:12:57.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3799: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2026-02-13T02:12:57.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:12:57.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:12:57.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:12:57.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:12:57.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2026-02-13T02:12:57.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:12:57.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:12:57.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:12:57.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:12:57.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:12:57.962 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:12:57.962 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:12:57.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:12:57.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:12:57.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:12:58.270 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:12:58.270 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2026-02-13T02:12:58.270 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:12:58.270 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:12:58.270 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:12:58.271 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:12:58.271 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:12:58.271 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2026-02-13T02:12:58.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-13T02:12:58.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:12:58.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:12:58.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:12:58.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:12:58.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:12:58.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:12:58.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:12:58.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:12:58.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:12:58.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:12:58.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:12:58.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:12:58.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:12:58.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:12:58.561 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:12:58.561 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:12:58.561 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:12:58.561 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:12:58.561 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:12:58.561 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:12:58.562 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:12:58.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:12:58.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:12:58.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:12:58.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:12:58.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:12:58.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:12:58.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:12:58.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:12:58.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:12:58.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:12:58.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:12:58.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:12:58.564 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:12:58.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:12:58.564 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:12:58.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:12:58.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:12:58.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:12:58.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:12:58.583 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:58.569+0000 7f86e1422a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:12:58.585 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:58.573+0000 7f86e1422a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:12:58.585 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:58.573+0000 7f86e1422a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:12:58.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:12:58.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:12:58.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:12:58.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:12:58.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:12:58.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:12:58.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:12:58.726 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:12:58.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:12:58.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:12:58.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:12:58.918 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:58.905+0000 7f86e1422a40 -1 Falling back to public interface 2026-02-13T02:12:59.563 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:12:59.549+0000 7f86e1422a40 -1 osd.1 86 log_to_monitors true 2026-02-13T02:12:59.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:12:59.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:12:59.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:12:59.891 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:12:59.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:12:59.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:13:00.052 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 90 up_thru 90 down_at 87 last_clean_interval [78,86) [v2:127.0.0.1:6810/742811890,v1:127.0.0.1:6811/742811890] [v2:127.0.0.1:6812/742811890,v1:127.0.0.1:6813/742811890] exists,up 4869dc97-b36a-441e-9ba9-92eb0a3698d2 2026-02-13T02:13:00.053 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:13:00.053 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:13:00.053 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:13:00.053 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:13:00.053 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:13:00.053 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:13:00.053 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:13:00.053 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:13:00.054 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:13:00.054 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:13:00.054 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:13:00.054 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:13:00.054 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:13:00.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:13:00.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:13:00.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:13:00.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:13:00.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:13:00.157 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:13:00.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:13:00.318 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:13:00.318 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:13:00.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:13:00.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:13:00.318 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:13:00.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=365072220164 2026-02-13T02:13:00.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 365072220164 2026-02-13T02:13:00.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-365072220164' 2026-02-13T02:13:00.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:13:00.389 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:13:00.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=386547056642 2026-02-13T02:13:00.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 386547056642 2026-02-13T02:13:00.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-365072220164 1-386547056642' 2026-02-13T02:13:00.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:13:00.462 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:13:00.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=304942678024 2026-02-13T02:13:00.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 304942678024 2026-02-13T02:13:00.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-365072220164 1-386547056642 2-304942678024' 2026-02-13T02:13:00.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:13:00.534 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-365072220164 2026-02-13T02:13:00.534 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:13:00.536 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:13:00.536 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-365072220164 2026-02-13T02:13:00.537 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:13:00.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=365072220164 2026-02-13T02:13:00.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 365072220164' 2026-02-13T02:13:00.538 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 365072220164 2026-02-13T02:13:00.539 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:13:00.704 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 365072220163 -lt 365072220164 2026-02-13T02:13:00.704 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:13:01.705 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:13:01.706 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:13:01.866 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 365072220163 -lt 365072220164 2026-02-13T02:13:01.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:13:02.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:13:02.868 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:13:03.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 365072220164 -lt 365072220164 2026-02-13T02:13:03.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:13:03.034 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-386547056642 2026-02-13T02:13:03.034 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:13:03.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:13:03.036 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-386547056642 2026-02-13T02:13:03.037 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:13:03.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=386547056642 2026-02-13T02:13:03.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 386547056642' 2026-02-13T02:13:03.038 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 386547056642 2026-02-13T02:13:03.039 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:13:03.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 386547056642 -lt 386547056642 2026-02-13T02:13:03.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:13:03.208 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-304942678024 2026-02-13T02:13:03.208 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:13:03.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:13:03.210 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-304942678024 2026-02-13T02:13:03.210 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:13:03.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=304942678024 2026-02-13T02:13:03.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 304942678024' 2026-02-13T02:13:03.212 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 304942678024 2026-02-13T02:13:03.212 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:13:03.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 304942678025 -lt 304942678024 2026-02-13T02:13:03.379 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:13:03.380 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:13:03.380 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:13:03.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:13:03.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:13:03.570 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:13:03.571 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:13:03.571 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:13:03.571 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:13:03.571 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:13:03.571 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:13:03.729 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:13:03.730 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:13:03.730 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:13:03.730 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:13:03.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:13:03.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:13:03.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:13:03.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-13T02:13:03.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ7 2026-02-13T02:13:03.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2026-02-13T02:13:03.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:13:03.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-13T02:13:03.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ7 2026-02-13T02:13:03.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:13:03.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:13:03.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:13:04.170 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:13:04.183 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:13:04.373 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:13:04.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:13:04.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:13:04.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:13:04.406 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 7 % 2 2026-02-13T02:13:04.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=1 2026-02-13T02:13:04.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-13T02:13:04.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3803: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2026-02-13T02:13:04.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3804: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2026-02-13T02:13:04.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3805: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2026-02-13T02:13:04.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3808: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2026-02-13T02:13:04.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:13:04.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:13:04.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T02:13:04.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:13:04.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2026-02-13T02:13:04.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:13:04.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:13:04.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T02:13:04.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:13:04.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T02:13:04.430 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:13:04.430 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:13:04.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:13:04.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:13:04.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:13:04.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:13:04.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2026-02-13T02:13:04.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:13:04.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:13:04.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:13:04.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:13:04.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:13:04.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2026-02-13T02:13:04.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-13T02:13:04.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:13:04.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:13:04.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:13:04.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:13:04.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:13:04.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:13:04.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:13:04.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:13:04.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:13:04.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:13:04.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:13:04.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:13:04.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:13:04.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:13:04.883 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:13:04.883 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:13:04.884 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:13:04.884 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:13:04.884 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:13:04.884 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:13:04.884 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:13:04.886 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:13:04.886 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:13:04.886 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:13:04.886 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:13:04.886 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:13:04.886 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:13:04.886 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:13:04.886 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:13:04.886 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:13:04.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:13:04.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:13:04.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:13:04.887 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T02:13:04.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:13:04.887 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T02:13:04.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:13:04.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:13:04.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:13:04.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:13:04.907 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:13:04.892+0000 7fe9e29bca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:13:04.908 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:13:04.896+0000 7fe9e29bca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:13:04.909 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:13:04.896+0000 7fe9e29bca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:13:05.049 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:13:05.049 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:13:05.049 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:13:05.049 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:13:05.049 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:13:05.049 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:13:05.049 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:13:05.050 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:13:05.050 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:13:05.050 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:13:05.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:13:05.254 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:13:05.240+0000 7fe9e29bca40 -1 Falling back to public interface 2026-02-13T02:13:05.794 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:13:05.780+0000 7fe9e29bca40 -1 osd.0 93 log_to_monitors true 2026-02-13T02:13:06.206 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:13:06.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:13:06.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:13:06.207 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:13:06.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:13:06.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:13:06.366 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 97 up_thru 97 down_at 94 last_clean_interval [85,93) [v2:127.0.0.1:6800/1269119298,v1:127.0.0.1:6801/1269119298] [v2:127.0.0.1:6802/1269119298,v1:127.0.0.1:6803/1269119298] exists,up a5bec03f-4e86-4fe9-8351-a3f10d7024e0 2026-02-13T02:13:06.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:13:06.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:13:06.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:13:06.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:13:06.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:13:06.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:13:06.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:13:06.367 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:13:06.367 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:13:06.368 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:13:06.368 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:13:06.368 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:13:06.368 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:13:06.469 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:13:06.469 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:13:06.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:13:06.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:13:06.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:13:06.470 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:13:06.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:13:06.640 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:13:06.640 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:13:06.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:13:06.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:13:06.640 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:13:06.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=416611827714 2026-02-13T02:13:06.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 416611827714 2026-02-13T02:13:06.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-416611827714' 2026-02-13T02:13:06.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:13:06.713 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:13:06.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=386547056644 2026-02-13T02:13:06.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 386547056644 2026-02-13T02:13:06.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-416611827714 1-386547056644' 2026-02-13T02:13:06.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:13:06.788 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:13:06.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=304942678026 2026-02-13T02:13:06.859 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 304942678026 2026-02-13T02:13:06.859 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-416611827714 1-386547056644 2-304942678026' 2026-02-13T02:13:06.859 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:13:06.859 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-416611827714 2026-02-13T02:13:06.859 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:13:06.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:13:06.862 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-416611827714 2026-02-13T02:13:06.862 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:13:06.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=416611827714 2026-02-13T02:13:06.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 416611827714' 2026-02-13T02:13:06.864 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 416611827714 2026-02-13T02:13:06.864 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:13:07.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 416611827714 2026-02-13T02:13:07.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:13:08.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:13:08.037 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:13:08.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 416611827714 2026-02-13T02:13:08.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:13:09.199 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:13:09.199 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:13:09.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 416611827714 -lt 416611827714 2026-02-13T02:13:09.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:13:09.366 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-386547056644 2026-02-13T02:13:09.366 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:13:09.368 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:13:09.368 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-386547056644 2026-02-13T02:13:09.369 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:13:09.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=386547056644 2026-02-13T02:13:09.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 386547056644' 2026-02-13T02:13:09.370 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 386547056644 2026-02-13T02:13:09.371 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:13:09.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 386547056644 -lt 386547056644 2026-02-13T02:13:09.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:13:09.543 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-304942678026 2026-02-13T02:13:09.543 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:13:09.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:13:09.545 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-304942678026 2026-02-13T02:13:09.546 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:13:09.547 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=304942678026 2026-02-13T02:13:09.547 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 304942678026' 2026-02-13T02:13:09.547 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 304942678026 2026-02-13T02:13:09.548 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:13:09.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 304942678027 -lt 304942678026 2026-02-13T02:13:09.713 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:13:09.714 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:13:09.714 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:13:09.906 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:13:09.906 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:13:09.907 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:13:09.907 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:13:09.907 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:13:09.907 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:13:09.907 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:13:09.907 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:13:10.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:13:10.061 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:13:10.062 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:13:10.062 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:13:10.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:13:10.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:13:10.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:13:10.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3809: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2026-02-13T02:13:10.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:13:10.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:13:10.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T02:13:10.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:13:10.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2026-02-13T02:13:10.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:13:10.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:13:10.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T02:13:10.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:13:10.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T02:13:10.263 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:13:10.263 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:13:10.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:13:10.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:13:10.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:13:10.572 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:13:10.572 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2026-02-13T02:13:10.572 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:13:10.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:13:10.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:13:10.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:13:10.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:13:10.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2026-02-13T02:13:10.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-13T02:13:10.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:13:10.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:13:10.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:13:10.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:13:10.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:13:10.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:13:10.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:13:10.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:13:10.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:13:10.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:13:10.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:13:10.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:13:10.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:13:10.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:13:10.861 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:13:10.861 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:13:10.861 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:13:10.861 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:13:10.861 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:13:10.861 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:13:10.863 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:13:10.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:13:10.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:13:10.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:13:10.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:13:10.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:13:10.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:13:10.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:13:10.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:13:10.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:13:10.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:13:10.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:13:10.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:13:10.864 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T02:13:10.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:13:10.864 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T02:13:10.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:13:10.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:13:10.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:13:10.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:13:10.885 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:13:10.872+0000 7f727214ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:13:10.887 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:13:10.872+0000 7f727214ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:13:10.888 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:13:10.872+0000 7f727214ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:13:11.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:13:11.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:13:11.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:13:11.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:13:11.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:13:11.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:13:11.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:13:11.030 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:13:11.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:13:11.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:13:11.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:13:11.350 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:13:11.336+0000 7f727214ea40 -1 Falling back to public interface 2026-02-13T02:13:11.950 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:13:11.936+0000 7f727214ea40 -1 osd.0 98 log_to_monitors true 2026-02-13T02:13:12.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:13:12.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:13:12.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:13:12.194 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:13:12.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:13:12.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:13:12.358 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 102 up_thru 102 down_at 99 last_clean_interval [97,98) [v2:127.0.0.1:6800/3865539018,v1:127.0.0.1:6801/3865539018] [v2:127.0.0.1:6802/3865539018,v1:127.0.0.1:6803/3865539018] exists,up a5bec03f-4e86-4fe9-8351-a3f10d7024e0 2026-02-13T02:13:12.359 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:13:12.359 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:13:12.359 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:13:12.359 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:13:12.359 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:13:12.359 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:13:12.359 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:13:12.359 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:13:12.360 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:13:12.360 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:13:12.360 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:13:12.360 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:13:12.360 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:13:12.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:13:12.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:13:12.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:13:12.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:13:12.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:13:12.478 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:13:12.642 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:13:12.642 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:13:12.643 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:13:12.643 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:13:12.643 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:13:12.643 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:13:12.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=438086664194 2026-02-13T02:13:12.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 438086664194 2026-02-13T02:13:12.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-438086664194' 2026-02-13T02:13:12.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:13:12.719 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:13:12.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=386547056646 2026-02-13T02:13:12.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 386547056646 2026-02-13T02:13:12.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-438086664194 1-386547056646' 2026-02-13T02:13:12.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:13:12.794 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:13:12.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=304942678029 2026-02-13T02:13:12.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 304942678029 2026-02-13T02:13:12.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-438086664194 1-386547056646 2-304942678029' 2026-02-13T02:13:12.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:13:12.864 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-438086664194 2026-02-13T02:13:12.864 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:13:12.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:13:12.866 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-438086664194 2026-02-13T02:13:12.866 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:13:12.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=438086664194 2026-02-13T02:13:12.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 438086664194' 2026-02-13T02:13:12.868 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 438086664194 2026-02-13T02:13:12.868 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:13:13.042 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 438086664194 2026-02-13T02:13:13.042 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:13:14.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:13:14.044 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:13:14.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 438086664194 2026-02-13T02:13:14.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:13:15.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:13:15.206 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:13:15.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 438086664194 -lt 438086664194 2026-02-13T02:13:15.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:13:15.371 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-386547056646 2026-02-13T02:13:15.371 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:13:15.372 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:13:15.373 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-386547056646 2026-02-13T02:13:15.373 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:13:15.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=386547056646 2026-02-13T02:13:15.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 386547056646' 2026-02-13T02:13:15.375 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 386547056646 2026-02-13T02:13:15.376 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:13:15.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 386547056646 -lt 386547056646 2026-02-13T02:13:15.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:13:15.543 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-304942678029 2026-02-13T02:13:15.543 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:13:15.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:13:15.545 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-304942678029 2026-02-13T02:13:15.545 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:13:15.547 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=304942678029 2026-02-13T02:13:15.547 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 304942678029' 2026-02-13T02:13:15.547 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 304942678029 2026-02-13T02:13:15.547 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:13:15.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 304942678029 -lt 304942678029 2026-02-13T02:13:15.721 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:13:15.722 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:13:15.722 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:13:15.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:13:15.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:13:15.920 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:13:15.920 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:13:15.920 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:13:15.921 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:13:15.921 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:13:15.921 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:13:16.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:13:16.081 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:13:16.082 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:13:16.082 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:13:16.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:13:16.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:13:16.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:13:16.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3810: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2026-02-13T02:13:16.276 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3816: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2026-02-13T02:13:16.276 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-13T02:13:16.276 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=EOBJ0 2026-02-13T02:13:16.276 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool EOBJ0 2026-02-13T02:13:16.276 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T02:13:16.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3816: corrupt_scrub_erasure: local pg=3.0 2026-02-13T02:13:16.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3818: corrupt_scrub_erasure: pg_scrub 3.0 2026-02-13T02:13:16.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=3.0 2026-02-13T02:13:16.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 3.0 2026-02-13T02:13:16.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-13T02:13:16.438 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-13T02:13:16.439 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:13:16.439 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:13:16.439 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:13:16.439 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:13:16.439 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:13:16.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-13T02:13:16.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-13T02:13:16.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-13T02:13:16.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-13T02:13:16.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:13:16.725 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:13:16.886 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:13:16.886 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:13:16.886 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:13:16.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:13:16.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:13:16.887 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:13:16.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=438086664195 2026-02-13T02:13:16.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 438086664195 2026-02-13T02:13:16.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-438086664195' 2026-02-13T02:13:16.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:13:16.962 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:13:17.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=386547056648 2026-02-13T02:13:17.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 386547056648 2026-02-13T02:13:17.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-438086664195 1-386547056648' 2026-02-13T02:13:17.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:13:17.036 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:13:17.108 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=304942678031 2026-02-13T02:13:17.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 304942678031 2026-02-13T02:13:17.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-438086664195 1-386547056648 2-304942678031' 2026-02-13T02:13:17.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:13:17.109 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-438086664195 2026-02-13T02:13:17.109 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:13:17.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:13:17.111 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-438086664195 2026-02-13T02:13:17.112 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:13:17.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=438086664195 2026-02-13T02:13:17.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 438086664195' 2026-02-13T02:13:17.114 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 438086664195 2026-02-13T02:13:17.114 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:13:17.271 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 438086664194 -lt 438086664195 2026-02-13T02:13:17.271 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:13:18.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:13:18.273 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:13:18.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 438086664196 -lt 438086664195 2026-02-13T02:13:18.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:13:18.437 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-386547056648 2026-02-13T02:13:18.437 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:13:18.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:13:18.439 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-386547056648 2026-02-13T02:13:18.439 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:13:18.441 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=386547056648 2026-02-13T02:13:18.441 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 386547056648' 2026-02-13T02:13:18.441 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 386547056648 2026-02-13T02:13:18.442 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:13:18.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 386547056648 -lt 386547056648 2026-02-13T02:13:18.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:13:18.603 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-304942678031 2026-02-13T02:13:18.603 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:13:18.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:13:18.605 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-304942678031 2026-02-13T02:13:18.606 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:13:18.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=304942678031 2026-02-13T02:13:18.607 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 304942678031 2026-02-13T02:13:18.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 304942678031' 2026-02-13T02:13:18.608 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:13:18.773 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 304942678031 -lt 304942678031 2026-02-13T02:13:18.773 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-13T02:13:18.773 INFO:tasks.workunit.client.0.trial112.stdout:#---------- 3.0 loop 0 2026-02-13T02:13:18.774 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-13T02:13:18.774 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-13T02:13:18.774 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-13T02:13:18.774 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-13T02:13:18.775 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-13T02:13:18.775 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-13T02:13:18.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-13T02:13:18.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-13T02:13:18.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-13T02:13:18.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-13T02:13:18.851 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: get_last_scrub_stamp 3.0 2026-02-13T02:13:18.851 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:13:18.851 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:13:18.851 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:13:18.852 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:13:19.006 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: local last_scrub=2026-02-13T02:12:04.260389+0000 2026-02-13T02:13:19.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_scrub: ceph pg scrub 3.0 2026-02-13T02:13:19.153 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 3.0s0 on osd.1 to scrub 2026-02-13T02:13:19.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_scrub: wait_for_scrub 3.0 2026-02-13T02:12:04.260389+0000 2026-02-13T02:13:19.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-13T02:13:19.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:12:04.260389+0000 2026-02-13T02:13:19.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:13:19.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:13:19.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:13:19.165 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-13T02:13:19.165 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:13:19.165 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:13:19.165 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:13:19.165 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:13:19.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:12:04.260389+0000 '>' 2026-02-13T02:12:04.260389+0000 2026-02-13T02:13:19.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:13:20.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:13:20.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:13:20.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-13T02:13:20.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:13:20.324 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:13:20.324 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:13:20.324 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:13:20.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:12:04.260389+0000 '>' 2026-02-13T02:12:04.260389+0000 2026-02-13T02:13:20.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:13:21.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:13:21.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:13:21.483 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-13T02:13:21.483 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:13:21.483 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:13:21.483 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:13:21.483 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:13:21.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:12:04.260389+0000 '>' 2026-02-13T02:12:04.260389+0000 2026-02-13T02:13:21.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:13:22.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:13:22.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:13:22.640 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-13T02:13:22.640 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:13:22.640 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:13:22.641 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:13:22.641 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:13:22.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:12:04.260389+0000 '>' 2026-02-13T02:12:04.260389+0000 2026-02-13T02:13:22.797 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:13:23.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:13:23.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:13:23.799 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-13T02:13:23.799 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:13:23.799 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:13:23.799 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:13:23.799 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:13:23.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:12:04.260389+0000 '>' 2026-02-13T02:12:04.260389+0000 2026-02-13T02:13:23.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:13:24.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:13:24.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:13:24.958 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-13T02:13:24.958 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:13:24.959 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:13:24.959 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:13:24.959 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:13:25.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:12:04.260389+0000 '>' 2026-02-13T02:12:04.260389+0000 2026-02-13T02:13:25.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:13:26.121 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:13:26.121 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:13:26.122 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-13T02:13:26.122 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:13:26.122 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:13:26.123 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:13:26.123 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:13:26.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:12:04.260389+0000 '>' 2026-02-13T02:12:04.260389+0000 2026-02-13T02:13:26.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:13:27.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:13:27.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:13:27.286 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-13T02:13:27.286 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:13:27.286 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:13:27.287 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:13:27.287 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:13:27.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:13:19.672879+0000 '>' 2026-02-13T02:12:04.260389+0000 2026-02-13T02:13:27.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:13:27.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3820: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2026-02-13T02:13:27.466 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3822: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2026-02-13T02:13:27.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3822: corrupt_scrub_erasure: test 1 = 1 2026-02-13T02:13:27.478 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3824: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2026-02-13T02:13:27.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3824: corrupt_scrub_erasure: test 3.0 = 3.0 2026-02-13T02:13:27.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3826: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2026-02-13T02:13:27.505 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3828: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2026-02-13T02:13:27.516 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3828: corrupt_scrub_erasure: epoch=102 2026-02-13T02:13:27.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3830: corrupt_scrub_erasure: jq 'def walk(f): 2026-02-13T02:13:27.517 INFO:tasks.workunit.client.0.trial112.stderr: . as $in 2026-02-13T02:13:27.517 INFO:tasks.workunit.client.0.trial112.stderr: | if type == "object" then 2026-02-13T02:13:27.518 INFO:tasks.workunit.client.0.trial112.stderr: reduce keys[] as $key 2026-02-13T02:13:27.518 INFO:tasks.workunit.client.0.trial112.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-13T02:13:27.518 INFO:tasks.workunit.client.0.trial112.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-13T02:13:27.518 INFO:tasks.workunit.client.0.trial112.stderr: else f 2026-02-13T02:13:27.518 INFO:tasks.workunit.client.0.trial112.stderr: end; 2026-02-13T02:13:27.518 INFO:tasks.workunit.client.0.trial112.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-13T02:13:27.518 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-13T02:13:27.518 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-13T02:13:27.518 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-13T02:13:27.518 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-13T02:13:27.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3830: corrupt_scrub_erasure: jq .inconsistents 2026-02-13T02:13:27.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3830: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-13T02:13:27.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4458: corrupt_scrub_erasure: jq 'def walk(f): 2026-02-13T02:13:27.535 INFO:tasks.workunit.client.0.trial112.stderr: . as $in 2026-02-13T02:13:27.535 INFO:tasks.workunit.client.0.trial112.stderr: | if type == "object" then 2026-02-13T02:13:27.535 INFO:tasks.workunit.client.0.trial112.stderr: reduce keys[] as $key 2026-02-13T02:13:27.535 INFO:tasks.workunit.client.0.trial112.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-13T02:13:27.535 INFO:tasks.workunit.client.0.trial112.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-13T02:13:27.535 INFO:tasks.workunit.client.0.trial112.stderr: else f 2026-02-13T02:13:27.535 INFO:tasks.workunit.client.0.trial112.stderr: end; 2026-02-13T02:13:27.535 INFO:tasks.workunit.client.0.trial112.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-13T02:13:27.535 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-13T02:13:27.535 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-13T02:13:27.536 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-13T02:13:27.536 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2026-02-13T02:13:27.536 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4458: corrupt_scrub_erasure: jq .inconsistents 2026-02-13T02:13:27.536 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4458: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-13T02:13:27.558 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4459: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-13T02:13:27.558 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2513: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-13T02:13:27.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4460: corrupt_scrub_erasure: test no = yes 2026-02-13T02:13:27.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4465: corrupt_scrub_erasure: test '' = yes 2026-02-13T02:13:27.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4470: corrupt_scrub_erasure: pg_deep_scrub 3.0 2026-02-13T02:13:27.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1932: pg_deep_scrub: local pgid=3.0 2026-02-13T02:13:27.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: pg_deep_scrub: wait_for_pg_clean 3.0 2026-02-13T02:13:27.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-13T02:13:27.560 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-13T02:13:27.560 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:13:27.561 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:13:27.561 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:13:27.561 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:13:27.561 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:13:27.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-13T02:13:27.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-13T02:13:27.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-13T02:13:27.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-13T02:13:27.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:13:27.862 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:13:28.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:13:28.018 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:13:28.019 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:13:28.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:13:28.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:13:28.019 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:13:28.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=438086664199 2026-02-13T02:13:28.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 438086664199 2026-02-13T02:13:28.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-438086664199' 2026-02-13T02:13:28.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:13:28.094 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:13:28.167 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=386547056651 2026-02-13T02:13:28.167 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 386547056651 2026-02-13T02:13:28.167 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-438086664199 1-386547056651' 2026-02-13T02:13:28.167 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:13:28.167 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:13:28.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=304942678034 2026-02-13T02:13:28.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 304942678034 2026-02-13T02:13:28.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-438086664199 1-386547056651 2-304942678034' 2026-02-13T02:13:28.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:13:28.241 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-438086664199 2026-02-13T02:13:28.241 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:13:28.242 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:13:28.243 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-438086664199 2026-02-13T02:13:28.243 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:13:28.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=438086664199 2026-02-13T02:13:28.245 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 438086664199 2026-02-13T02:13:28.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 438086664199' 2026-02-13T02:13:28.246 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:13:28.402 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 438086664199 -lt 438086664199 2026-02-13T02:13:28.402 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:13:28.403 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-386547056651 2026-02-13T02:13:28.403 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:13:28.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:13:28.405 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-386547056651 2026-02-13T02:13:28.405 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:13:28.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=386547056651 2026-02-13T02:13:28.407 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 386547056651 2026-02-13T02:13:28.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 386547056651' 2026-02-13T02:13:28.408 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:13:28.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 386547056651 -lt 386547056651 2026-02-13T02:13:28.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:13:28.580 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-304942678034 2026-02-13T02:13:28.580 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:13:28.581 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:13:28.582 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-304942678034 2026-02-13T02:13:28.582 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:13:28.584 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=304942678034 2026-02-13T02:13:28.584 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 304942678034 2026-02-13T02:13:28.584 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 304942678034' 2026-02-13T02:13:28.584 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:13:28.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 304942678034 -lt 304942678034 2026-02-13T02:13:28.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-13T02:13:28.754 INFO:tasks.workunit.client.0.trial112.stdout:#---------- 3.0 loop 0 2026-02-13T02:13:28.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-13T02:13:28.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-13T02:13:28.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-13T02:13:28.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-13T02:13:28.756 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-13T02:13:28.756 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-13T02:13:28.827 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean+inconsistent 2026-02-13T02:13:28.827 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean+inconsistent == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-13T02:13:28.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-13T02:13:28.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-13T02:13:28.828 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-13T02:13:28.828 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:13:28.828 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:13:28.828 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:13:28.828 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-13T02:13:28.981 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: local last_scrub=2026-02-13T02:12:04.260389+0000 2026-02-13T02:13:28.981 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: pg_deep_scrub: ceph pg deep-scrub 3.0 2026-02-13T02:13:29.128 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2026-02-13T02:13:29.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: pg_deep_scrub: wait_for_scrub 3.0 2026-02-13T02:12:04.260389+0000 last_deep_scrub_stamp 2026-02-13T02:13:29.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-13T02:13:29.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:12:04.260389+0000 2026-02-13T02:13:29.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_deep_scrub_stamp 2026-02-13T02:13:29.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:13:29.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:13:29.141 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-13T02:13:29.141 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:13:29.141 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:13:29.142 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:13:29.142 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-13T02:13:29.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:12:04.260389+0000 '>' 2026-02-13T02:12:04.260389+0000 2026-02-13T02:13:29.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:13:30.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:13:30.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:13:30.299 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-13T02:13:30.299 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:13:30.299 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:13:30.299 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:13:30.299 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-13T02:13:30.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:12:04.260389+0000 '>' 2026-02-13T02:12:04.260389+0000 2026-02-13T02:13:30.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:13:31.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:13:31.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:13:31.460 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-13T02:13:31.460 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:13:31.460 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:13:31.461 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:13:31.461 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-13T02:13:31.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:12:04.260389+0000 '>' 2026-02-13T02:12:04.260389+0000 2026-02-13T02:13:31.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:13:32.624 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:13:32.625 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:13:32.625 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-13T02:13:32.625 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:13:32.625 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:13:32.626 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:13:32.626 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-13T02:13:32.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:12:04.260389+0000 '>' 2026-02-13T02:12:04.260389+0000 2026-02-13T02:13:32.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:13:33.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:13:33.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:13:33.793 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-13T02:13:33.793 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:13:33.794 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:13:33.794 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:13:33.794 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-13T02:13:33.959 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:12:04.260389+0000 '>' 2026-02-13T02:12:04.260389+0000 2026-02-13T02:13:33.959 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:13:34.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:13:34.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:13:34.961 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-13T02:13:34.961 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:13:34.961 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:13:34.961 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-13T02:13:34.962 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:13:35.123 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:12:04.260389+0000 '>' 2026-02-13T02:12:04.260389+0000 2026-02-13T02:13:35.123 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:13:36.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:13:36.125 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:13:36.125 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-13T02:13:36.126 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:13:36.126 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:13:36.126 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:13:36.126 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-13T02:13:36.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:12:04.260389+0000 '>' 2026-02-13T02:12:04.260389+0000 2026-02-13T02:13:36.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:13:37.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:13:37.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:13:37.285 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-13T02:13:37.285 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:13:37.285 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:13:37.285 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:13:37.285 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-13T02:13:37.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:13:29.691302+0000 '>' 2026-02-13T02:12:04.260389+0000 2026-02-13T02:13:37.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:13:37.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4472: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2026-02-13T02:13:37.465 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4474: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2026-02-13T02:13:37.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4474: corrupt_scrub_erasure: test 1 = 1 2026-02-13T02:13:37.479 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4476: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2026-02-13T02:13:37.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4476: corrupt_scrub_erasure: test 3.0 = 3.0 2026-02-13T02:13:37.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4478: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2026-02-13T02:13:37.508 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4479: corrupt_scrub_erasure: cp td/osd-scrub-repair/json /tmp/rrr6.json 2026-02-13T02:13:37.509 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4481: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2026-02-13T02:13:37.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4481: corrupt_scrub_erasure: epoch=102 2026-02-13T02:13:37.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4483: corrupt_scrub_erasure: '[' true = true ']' 2026-02-13T02:13:37.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4485: corrupt_scrub_erasure: jq 'def walk(f): 2026-02-13T02:13:37.523 INFO:tasks.workunit.client.0.trial112.stderr: . as $in 2026-02-13T02:13:37.523 INFO:tasks.workunit.client.0.trial112.stderr: | if type == "object" then 2026-02-13T02:13:37.523 INFO:tasks.workunit.client.0.trial112.stderr: reduce keys[] as $key 2026-02-13T02:13:37.523 INFO:tasks.workunit.client.0.trial112.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-13T02:13:37.523 INFO:tasks.workunit.client.0.trial112.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-13T02:13:37.523 INFO:tasks.workunit.client.0.trial112.stderr: else f 2026-02-13T02:13:37.523 INFO:tasks.workunit.client.0.trial112.stderr: end; 2026-02-13T02:13:37.523 INFO:tasks.workunit.client.0.trial112.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-13T02:13:37.523 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-13T02:13:37.523 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-13T02:13:37.523 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-13T02:13:37.523 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-13T02:13:37.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4485: corrupt_scrub_erasure: jq .inconsistents 2026-02-13T02:13:37.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4485: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-13T02:13:37.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5890: corrupt_scrub_erasure: jq 'def walk(f): 2026-02-13T02:13:37.542 INFO:tasks.workunit.client.0.trial112.stderr: . as $in 2026-02-13T02:13:37.542 INFO:tasks.workunit.client.0.trial112.stderr: | if type == "object" then 2026-02-13T02:13:37.542 INFO:tasks.workunit.client.0.trial112.stderr: reduce keys[] as $key 2026-02-13T02:13:37.542 INFO:tasks.workunit.client.0.trial112.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-13T02:13:37.542 INFO:tasks.workunit.client.0.trial112.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-13T02:13:37.543 INFO:tasks.workunit.client.0.trial112.stderr: else f 2026-02-13T02:13:37.543 INFO:tasks.workunit.client.0.trial112.stderr: end; 2026-02-13T02:13:37.543 INFO:tasks.workunit.client.0.trial112.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-13T02:13:37.543 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-13T02:13:37.543 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-13T02:13:37.543 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-13T02:13:37.543 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2026-02-13T02:13:37.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5890: corrupt_scrub_erasure: jq .inconsistents 2026-02-13T02:13:37.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5890: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-13T02:13:37.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5891: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-13T02:13:37.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2513: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-13T02:13:37.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5892: corrupt_scrub_erasure: test no = yes 2026-02-13T02:13:37.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5903: corrupt_scrub_erasure: test '' = yes 2026-02-13T02:13:37.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5908: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2026-02-13T02:13:37.771 INFO:tasks.workunit.client.0.trial112.stderr:pool 'ecpool' removed 2026-02-13T02:13:37.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-13T02:13:37.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:13:37.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:13:37.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:13:37.782 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:13:37.783 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:13:37.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:13:37.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:13:37.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:13:37.893 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:13:37.894 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:13:37.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:13:37.895 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:13:37.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:13:37.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:13:37.897 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:13:37.899 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:13:37.899 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:13:37.899 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:13:37.900 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:13:37.901 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:13:37.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:13:37.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:13:37.928 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:13:37.929 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:13:37.929 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:13:37.929 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:13:37.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:13:37.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:13:37.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-13T02:13:37.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-13T02:13:37.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-13T02:13:37.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-13T02:13:37.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:13:37.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:13:37.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:13:37.932 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:13:37.932 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:13:37.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:13:37.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:13:37.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:13:37.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:13:37.934 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:13:37.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:13:37.935 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:13:37.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:13:37.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:13:37.937 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:13:37.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:13:37.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:13:37.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:13:37.939 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:13:37.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:13:37.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:13:37.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:13:37.943 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:13:37.943 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:13:37.943 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:13:37.944 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:13:37.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:13:37.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:13:37.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-13T02:13:37.946 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T02:13:37.947 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:13:37.947 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:13:37.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.35207 2026-02-13T02:13:37.949 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T02:13:37.950 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-13T02:13:37.950 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T02:13:37.950 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-13T02:13:37.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_replicated td/osd-scrub-repair 2026-02-13T02:13:37.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: local dir=td/osd-scrub-repair 2026-02-13T02:13:37.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1051: TEST_corrupt_scrub_replicated: local poolname=csr_pool 2026-02-13T02:13:37.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1052: TEST_corrupt_scrub_replicated: local total_objs=19 2026-02-13T02:13:37.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1054: TEST_corrupt_scrub_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2026-02-13T02:13:37.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-13T02:13:37.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-13T02:13:37.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-13T02:13:37.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-13T02:13:37.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-13T02:13:37.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2026-02-13T02:13:37.972 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-13T02:13:37.973 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:13:37.973 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:13:37.974 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:13:37.974 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:13:37.974 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:13:37.974 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:13:37.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2026-02-13T02:13:38.004 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-13T02:13:38.005 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-13T02:13:38.005 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:13:38.005 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:13:38.005 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-13T02:13:38.006 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-13T02:13:38.006 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:13:38.006 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:13:38.006 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:13:38.007 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:13:38.007 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:13:38.007 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:13:38.007 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:13:38.007 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:13:38.008 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get fsid 2026-02-13T02:13:38.064 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-13T02:13:38.064 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:13:38.065 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:13:38.065 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-13T02:13:38.066 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-13T02:13:38.066 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:13:38.066 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:13:38.066 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:13:38.066 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:13:38.067 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:13:38.067 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:13:38.067 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:13:38.067 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:13:38.067 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get mon_host 2026-02-13T02:13:38.123 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1055: TEST_corrupt_scrub_replicated: run_mgr td/osd-scrub-repair x 2026-02-13T02:13:38.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-13T02:13:38.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-13T02:13:38.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-13T02:13:38.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-13T02:13:38.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-13T02:13:38.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-13T02:13:38.223 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-13T02:13:38.223 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:13:38.223 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:13:38.223 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:13:38.223 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:13:38.223 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:13:38.223 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:13:38.224 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:13:38.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:13:38.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1056: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 0 2026-02-13T02:13:38.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:13:38.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:13:38.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-13T02:13:38.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:13:38.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:13:38.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:13:38.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:13:38.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:13:38.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:13:38.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:13:38.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:13:38.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:13:38.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:13:38.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:13:38.244 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:13:38.244 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:13:38.244 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:13:38.245 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:13:38.245 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:13:38.245 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:13:38.245 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:13:38.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:13:38.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:13:38.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:13:38.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:13:38.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:13:38.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:13:38.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:13:38.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:13:38.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:13:38.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:13:38.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:13:38.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:13:38.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:13:38.247 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:13:38.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=926506f0-916f-45ab-94cb-3aa841154594 2026-02-13T02:13:38.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 926506f0-916f-45ab-94cb-3aa841154594' 2026-02-13T02:13:38.248 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 926506f0-916f-45ab-94cb-3aa841154594 2026-02-13T02:13:38.248 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:13:38.258 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDSiI5p+jLIDhAAcCboKiC1RVtAKh+1yM2gXg== 2026-02-13T02:13:38.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDSiI5p+jLIDhAAcCboKiC1RVtAKh+1yM2gXg=="}' 2026-02-13T02:13:38.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 926506f0-916f-45ab-94cb-3aa841154594 -i td/osd-scrub-repair/0/new.json 2026-02-13T02:13:38.347 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:13:38.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-13T02:13:38.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDSiI5p+jLIDhAAcCboKiC1RVtAKh+1yM2gXg== --osd-uuid 926506f0-916f-45ab-94cb-3aa841154594 2026-02-13T02:13:38.372 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:13:38.356+0000 7fa82960ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:13:38.373 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:13:38.360+0000 7fa82960ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:13:38.374 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:13:38.360+0000 7fa82960ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:13:38.374 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:13:38.360+0000 7fa82960ea40 -1 bdev(0x55f95b206c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-13T02:13:38.374 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:13:38.360+0000 7fa82960ea40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-13T02:13:39.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-13T02:13:39.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:13:39.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-13T02:13:39.081 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T02:13:39.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:13:39.176 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T02:13:39.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-13T02:13:39.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:13:39.178 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:13:39.178 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:13:39.178 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:13:39.191 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:13:39.176+0000 7f37dbca7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:13:39.193 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:13:39.180+0000 7f37dbca7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:13:39.194 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:13:39.180+0000 7f37dbca7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:13:39.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-13T02:13:39.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:13:39.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:13:39.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:13:39.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:13:39.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:13:39.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:13:39.276 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:13:39.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:13:39.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:13:39.368 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:13:39.587 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:13:39.572+0000 7f37dbca7a40 -1 Falling back to public interface 2026-02-13T02:13:40.274 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:13:40.260+0000 7f37dbca7a40 -1 osd.0 0 log_to_monitors true 2026-02-13T02:13:40.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:13:40.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:13:40.371 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:13:40.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:13:40.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:13:40.372 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:13:40.494 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:13:41.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:13:41.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:13:41.496 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:13:41.497 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:13:41.497 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:13:41.497 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:13:41.660 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/651318686,v1:127.0.0.1:6801/651318686] [v2:127.0.0.1:6802/651318686,v1:127.0.0.1:6803/651318686] exists,up 926506f0-916f-45ab-94cb-3aa841154594 2026-02-13T02:13:41.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:13:41.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:13:41.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:13:41.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1057: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 1 2026-02-13T02:13:41.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:13:41.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:13:41.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-13T02:13:41.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:13:41.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:13:41.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:13:41.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:13:41.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:13:41.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:13:41.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:13:41.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:13:41.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:13:41.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:13:41.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:13:41.665 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:13:41.665 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:13:41.665 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:13:41.665 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:13:41.666 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:13:41.666 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:13:41.666 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:13:41.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:13:41.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:13:41.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:13:41.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:13:41.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:13:41.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:13:41.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:13:41.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:13:41.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:13:41.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:13:41.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:13:41.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:13:41.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:13:41.667 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:13:41.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=cfe9e8e4-e309-485e-8178-7cb3d645195b 2026-02-13T02:13:41.668 INFO:tasks.workunit.client.0.trial112.stdout:add osd1 cfe9e8e4-e309-485e-8178-7cb3d645195b 2026-02-13T02:13:41.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 cfe9e8e4-e309-485e-8178-7cb3d645195b' 2026-02-13T02:13:41.668 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:13:41.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDViI5pAeqxJxAAf5ilf2UPmejS9S0T1dR1YQ== 2026-02-13T02:13:41.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDViI5pAeqxJxAAf5ilf2UPmejS9S0T1dR1YQ=="}' 2026-02-13T02:13:41.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new cfe9e8e4-e309-485e-8178-7cb3d645195b -i td/osd-scrub-repair/1/new.json 2026-02-13T02:13:41.834 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:13:41.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-13T02:13:41.845 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDViI5pAeqxJxAAf5ilf2UPmejS9S0T1dR1YQ== --osd-uuid cfe9e8e4-e309-485e-8178-7cb3d645195b 2026-02-13T02:13:41.862 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:13:41.848+0000 7fbdb8fdfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:13:41.864 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:13:41.848+0000 7fbdb8fdfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:13:41.865 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:13:41.852+0000 7fbdb8fdfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:13:41.866 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:13:41.852+0000 7fbdb8fdfa40 -1 bdev(0x560547a10c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-13T02:13:41.866 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:13:41.852+0000 7fbdb8fdfa40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-13T02:13:42.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-13T02:13:42.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:13:42.497 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-13T02:13:42.497 INFO:tasks.workunit.client.0.trial112.stdout:adding osd1 key to auth repository 2026-02-13T02:13:42.498 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:13:42.705 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T02:13:42.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-13T02:13:42.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:13:42.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:13:42.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:13:42.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:13:42.729 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:13:42.716+0000 7faf2760ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:13:42.732 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:13:42.716+0000 7faf2760ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:13:42.732 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:13:42.716+0000 7faf2760ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:13:42.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-13T02:13:42.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:13:42.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:13:42.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:13:42.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:13:42.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:13:42.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:13:42.863 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:13:42.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:13:42.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:13:43.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:13:43.122 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:13:43.108+0000 7faf2760ea40 -1 Falling back to public interface 2026-02-13T02:13:43.715 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:13:43.700+0000 7faf2760ea40 -1 osd.1 0 log_to_monitors true 2026-02-13T02:13:44.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:13:44.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:13:44.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:13:44.029 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:13:44.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:13:44.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:13:44.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:13:45.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:13:45.195 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:13:45.195 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:13:45.195 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:13:45.196 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:13:45.196 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:13:45.359 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2588702924,v1:127.0.0.1:6811/2588702924] [v2:127.0.0.1:6812/2588702924,v1:127.0.0.1:6813/2588702924] exists,up cfe9e8e4-e309-485e-8178-7cb3d645195b 2026-02-13T02:13:45.359 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:13:45.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:13:45.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:13:45.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1058: TEST_corrupt_scrub_replicated: create_rbd_pool 2026-02-13T02:13:45.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-13T02:13:45.511 INFO:tasks.workunit.client.0.trial112.stderr:pool 'rbd' does not exist 2026-02-13T02:13:45.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-13T02:13:45.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-13T02:13:45.747 INFO:tasks.workunit.client.0.trial112.stderr:pool 'rbd' created 2026-02-13T02:13:45.760 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:13:46.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-13T02:13:47.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1059: TEST_corrupt_scrub_replicated: wait_for_clean 2026-02-13T02:13:47.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:13:47.368 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:13:47.368 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:13:47.368 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:13:47.369 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:13:47.369 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:13:47.369 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:13:47.369 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:13:47.369 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:13:47.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:13:47.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:13:47.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:13:47.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:13:47.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:13:47.484 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:13:47.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:13:47.648 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:13:47.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:13:47.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:13:47.649 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:13:47.720 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-13T02:13:47.720 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-13T02:13:47.720 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-13T02:13:47.720 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:13:47.721 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:13:47.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705666 2026-02-13T02:13:47.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705666 2026-02-13T02:13:47.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705666' 2026-02-13T02:13:47.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:13:47.795 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-13T02:13:47.795 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:13:47.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:13:47.797 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-13T02:13:47.797 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:13:47.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-13T02:13:47.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-13T02:13:47.799 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836483 2026-02-13T02:13:47.799 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:13:47.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-13T02:13:47.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:13:48.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:13:48.965 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:13:49.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-13T02:13:49.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:13:49.133 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705666 2026-02-13T02:13:49.134 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:13:49.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:13:49.136 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705666 2026-02-13T02:13:49.136 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:13:49.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705666 2026-02-13T02:13:49.138 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705666 2026-02-13T02:13:49.138 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705666' 2026-02-13T02:13:49.138 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:13:49.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705666 -lt 38654705666 2026-02-13T02:13:49.297 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:13:49.298 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:13:49.298 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:13:49.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-13T02:13:49.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:13:49.492 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:13:49.493 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:13:49.493 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:13:49.493 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:13:49.493 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:13:49.493 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:13:49.645 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-13T02:13:49.646 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:13:49.646 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:13:49.646 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:13:49.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-13T02:13:49.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:13:49.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:13:49.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1061: TEST_corrupt_scrub_replicated: create_pool foo 1 2026-02-13T02:13:49.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 1 2026-02-13T02:13:50.051 INFO:tasks.workunit.client.0.trial112.stderr:pool 'foo' created 2026-02-13T02:13:50.064 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:13:51.065 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1062: TEST_corrupt_scrub_replicated: create_pool csr_pool 1 1 2026-02-13T02:13:51.066 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create csr_pool 1 1 2026-02-13T02:13:51.279 INFO:tasks.workunit.client.0.trial112.stderr:pool 'csr_pool' created 2026-02-13T02:13:51.290 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:13:52.291 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1063: TEST_corrupt_scrub_replicated: wait_for_clean 2026-02-13T02:13:52.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:13:52.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:13:52.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:13:52.292 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:13:52.293 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:13:52.293 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:13:52.294 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:13:52.294 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:13:52.294 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:13:52.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:13:52.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:13:52.401 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:13:52.401 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:13:52.401 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:13:52.401 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:13:52.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:13:52.570 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:13:52.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:13:52.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:13:52.570 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:13:52.644 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836485 2026-02-13T02:13:52.644 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836485 2026-02-13T02:13:52.645 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485' 2026-02-13T02:13:52.645 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:13:52.645 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:13:52.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705668 2026-02-13T02:13:52.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705668 2026-02-13T02:13:52.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-38654705668' 2026-02-13T02:13:52.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:13:52.717 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836485 2026-02-13T02:13:52.717 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:13:52.718 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:13:52.719 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836485 2026-02-13T02:13:52.719 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:13:52.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836485 2026-02-13T02:13:52.721 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836485 2026-02-13T02:13:52.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-13T02:13:52.722 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:13:52.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836485 2026-02-13T02:13:52.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:13:53.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:13:53.897 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:13:54.053 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836485 2026-02-13T02:13:54.053 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:13:55.054 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:13:55.055 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:13:55.219 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 21474836485 2026-02-13T02:13:55.219 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:13:55.219 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705668 2026-02-13T02:13:55.219 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:13:55.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:13:55.222 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705668 2026-02-13T02:13:55.222 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:13:55.224 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705668 2026-02-13T02:13:55.224 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705668' 2026-02-13T02:13:55.224 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705668 2026-02-13T02:13:55.225 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:13:55.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705668 -lt 38654705668 2026-02-13T02:13:55.392 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:13:55.392 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:13:55.392 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:13:55.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:13:55.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:13:55.593 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:13:55.593 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:13:55.593 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:13:55.593 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:13:55.593 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:13:55.593 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:13:55.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:13:55.756 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:13:55.756 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:13:55.756 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:13:55.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:13:55.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:13:55.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:13:55.952 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: seq 1 19 2026-02-13T02:13:55.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-13T02:13:55.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ1 2026-02-13T02:13:55.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ1 2026-02-13T02:13:55.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:13:55.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-13T02:13:55.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ1 2026-02-13T02:13:55.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:13:55.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:13:55.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:13:56.161 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:13:56.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:13:56.366 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:13:56.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:13:56.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:13:56.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:13:56.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2026-02-13T02:13:56.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2026-02-13T02:13:56.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-13T02:13:56.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ2 2026-02-13T02:13:56.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ2 2026-02-13T02:13:56.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:13:56.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-13T02:13:56.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ2 2026-02-13T02:13:56.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:13:56.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:13:56.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:13:56.669 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:13:56.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:13:56.872 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:13:56.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:13:56.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:13:56.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:13:56.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2026-02-13T02:13:56.928 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2026-02-13T02:13:56.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-13T02:13:56.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ3 2026-02-13T02:13:56.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ3 2026-02-13T02:13:56.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:13:56.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-13T02:13:56.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ3 2026-02-13T02:13:56.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:13:56.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:13:56.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:13:57.167 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:13:57.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:13:57.372 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:13:57.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:13:57.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:13:57.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ3 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:13:57.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ3 hdr-ROBJ3 2026-02-13T02:13:57.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ3 key-ROBJ3 val-ROBJ3 2026-02-13T02:13:57.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-13T02:13:57.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ4 2026-02-13T02:13:57.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ4 2026-02-13T02:13:57.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:13:57.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-13T02:13:57.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ4 2026-02-13T02:13:57.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:13:57.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:13:57.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:13:57.672 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:13:57.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:13:57.876 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:13:57.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:13:57.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:13:57.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ4 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:13:57.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ4 hdr-ROBJ4 2026-02-13T02:13:57.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ4 key-ROBJ4 val-ROBJ4 2026-02-13T02:13:57.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-13T02:13:57.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ5 2026-02-13T02:13:57.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ5 2026-02-13T02:13:57.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:13:57.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-13T02:13:57.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ5 2026-02-13T02:13:57.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:13:57.959 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:13:57.959 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:13:58.170 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:13:58.183 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:13:58.376 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:13:58.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:13:58.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:13:58.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ5 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:13:58.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ5 hdr-ROBJ5 2026-02-13T02:13:58.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ5 key-ROBJ5 val-ROBJ5 2026-02-13T02:13:58.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-13T02:13:58.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ6 2026-02-13T02:13:58.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ6 2026-02-13T02:13:58.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:13:58.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-13T02:13:58.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ6 2026-02-13T02:13:58.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:13:58.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:13:58.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:13:58.680 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:13:58.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:13:58.882 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:13:58.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:13:58.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:13:58.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ6 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:13:58.915 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ6 hdr-ROBJ6 2026-02-13T02:13:58.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ6 key-ROBJ6 val-ROBJ6 2026-02-13T02:13:58.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-13T02:13:58.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ7 2026-02-13T02:13:58.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ7 2026-02-13T02:13:58.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:13:58.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-13T02:13:58.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ7 2026-02-13T02:13:58.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:13:58.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:13:58.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:13:59.176 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:13:59.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:13:59.381 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:13:59.393 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:13:59.394 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:13:59.394 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ7 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:13:59.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ7 hdr-ROBJ7 2026-02-13T02:13:59.439 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ7 key-ROBJ7 val-ROBJ7 2026-02-13T02:13:59.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-13T02:13:59.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ8 2026-02-13T02:13:59.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ8 2026-02-13T02:13:59.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:13:59.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-13T02:13:59.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ8 2026-02-13T02:13:59.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:13:59.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:13:59.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:13:59.666 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:13:59.680 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:13:59.871 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:13:59.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:13:59.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:13:59.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ8 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:13:59.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ8 hdr-ROBJ8 2026-02-13T02:13:59.929 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ8 key-ROBJ8 val-ROBJ8 2026-02-13T02:13:59.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-13T02:13:59.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ9 2026-02-13T02:13:59.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ9 2026-02-13T02:13:59.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:13:59.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-13T02:13:59.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ9 2026-02-13T02:13:59.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:13:59.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:13:59.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:14:00.164 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:14:00.175 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:14:00.369 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:14:00.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:14:00.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:14:00.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:14:00.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ9 hdr-ROBJ9 2026-02-13T02:14:00.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ9 key-ROBJ9 val-ROBJ9 2026-02-13T02:14:00.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-13T02:14:00.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ10 2026-02-13T02:14:00.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ10 2026-02-13T02:14:00.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:14:00.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-13T02:14:00.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ10 2026-02-13T02:14:00.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:14:00.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:14:00.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:14:00.672 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:14:00.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:14:00.875 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:14:00.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:14:00.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:14:00.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ10 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:14:00.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ10 hdr-ROBJ10 2026-02-13T02:14:00.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key-ROBJ10 val-ROBJ10 2026-02-13T02:14:00.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-13T02:14:00.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ11 2026-02-13T02:14:00.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ11 2026-02-13T02:14:00.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:14:00.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-13T02:14:00.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ11 2026-02-13T02:14:00.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:14:00.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:14:00.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:14:01.167 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:14:01.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:14:01.372 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:14:01.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:14:01.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:14:01.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ11 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:14:01.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ11 hdr-ROBJ11 2026-02-13T02:14:01.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ11 key-ROBJ11 val-ROBJ11 2026-02-13T02:14:01.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-13T02:14:01.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ12 2026-02-13T02:14:01.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ12 2026-02-13T02:14:01.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:14:01.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-13T02:14:01.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ12 2026-02-13T02:14:01.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:14:01.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:14:01.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:14:01.666 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:14:01.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:14:01.871 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:14:01.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:14:01.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:14:01.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ12 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:14:01.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ12 hdr-ROBJ12 2026-02-13T02:14:01.926 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ12 key-ROBJ12 val-ROBJ12 2026-02-13T02:14:01.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-13T02:14:01.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ13 2026-02-13T02:14:01.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ13 2026-02-13T02:14:01.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:14:01.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-13T02:14:01.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ13 2026-02-13T02:14:01.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:14:01.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:14:01.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:14:02.160 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:14:02.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:14:02.365 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:14:02.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:14:02.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:14:02.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ13 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:14:02.397 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ13 hdr-ROBJ13 2026-02-13T02:14:02.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ13 key-ROBJ13 val-ROBJ13 2026-02-13T02:14:02.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-13T02:14:02.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ14 2026-02-13T02:14:02.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ14 2026-02-13T02:14:02.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:14:02.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-13T02:14:02.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ14 2026-02-13T02:14:02.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:14:02.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:14:02.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:14:02.668 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:14:02.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:14:02.871 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:14:02.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:14:02.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:14:02.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ14 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:14:02.906 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ14 hdr-ROBJ14 2026-02-13T02:14:02.925 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ14 key-ROBJ14 val-ROBJ14 2026-02-13T02:14:02.950 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-13T02:14:02.950 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ15 2026-02-13T02:14:02.950 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ15 2026-02-13T02:14:02.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:14:02.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-13T02:14:02.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ15 2026-02-13T02:14:02.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:14:02.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:14:02.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:14:03.174 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:14:03.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:14:03.378 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:14:03.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:14:03.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:14:03.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ15 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:14:03.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ15 hdr-ROBJ15 2026-02-13T02:14:03.435 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ15 key-ROBJ15 val-ROBJ15 2026-02-13T02:14:03.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-13T02:14:03.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ16 2026-02-13T02:14:03.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ16 2026-02-13T02:14:03.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:14:03.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-13T02:14:03.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ16 2026-02-13T02:14:03.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:14:03.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:14:03.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:14:03.659 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:14:03.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:14:03.864 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:14:03.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:14:03.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:14:03.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ16 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:14:03.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ16 hdr-ROBJ16 2026-02-13T02:14:03.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ16 key-ROBJ16 val-ROBJ16 2026-02-13T02:14:03.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-13T02:14:03.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ17 2026-02-13T02:14:03.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ17 2026-02-13T02:14:03.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:14:03.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-13T02:14:03.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ17 2026-02-13T02:14:03.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:14:03.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:14:03.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:14:04.160 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:14:04.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:14:04.364 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:14:04.376 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:14:04.376 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:14:04.376 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ17 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:14:04.395 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ17 hdr-ROBJ17 2026-02-13T02:14:04.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ17 key-ROBJ17 val-ROBJ17 2026-02-13T02:14:04.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-13T02:14:04.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ18 2026-02-13T02:14:04.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ18 2026-02-13T02:14:04.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:14:04.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-13T02:14:04.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ18 2026-02-13T02:14:04.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:14:04.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:14:04.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:14:04.667 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:14:04.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:14:04.870 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:14:04.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:14:04.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:14:04.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ18 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:14:04.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ18 hdr-ROBJ18 2026-02-13T02:14:04.926 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ18 key-ROBJ18 val-ROBJ18 2026-02-13T02:14:04.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-13T02:14:04.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ19 2026-02-13T02:14:04.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ19 2026-02-13T02:14:04.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:14:04.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-13T02:14:04.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ19 2026-02-13T02:14:04.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:14:04.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:14:04.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:14:05.159 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:14:05.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:14:05.363 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:14:05.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:14:05.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:14:05.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:14:05.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ19 hdr-ROBJ19 2026-02-13T02:14:05.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ19 key-ROBJ19 val-ROBJ19 2026-02-13T02:14:05.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1074: TEST_corrupt_scrub_replicated: dd if=/dev/zero of=td/osd-scrub-repair/new.ROBJ19 bs=1024 count=1025 2026-02-13T02:14:05.448 INFO:tasks.workunit.client.0.trial112.stderr:1025+0 records in 2026-02-13T02:14:05.448 INFO:tasks.workunit.client.0.trial112.stderr:1025+0 records out 2026-02-13T02:14:05.448 INFO:tasks.workunit.client.0.trial112.stderr:1049600 bytes (1.0 MB, 1.0 MiB) copied, 0.00330813 s, 317 MB/s 2026-02-13T02:14:05.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1075: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/new.ROBJ19 2026-02-13T02:14:05.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1076: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ19 2026-02-13T02:14:05.477 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1078: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2026-02-13T02:14:05.477 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=csr_pool 2026-02-13T02:14:05.478 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=ROBJ0 2026-02-13T02:14:05.478 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map csr_pool ROBJ0 2026-02-13T02:14:05.478 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T02:14:05.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1078: TEST_corrupt_scrub_replicated: local pg=3.0 2026-02-13T02:14:05.638 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1079: TEST_corrupt_scrub_replicated: get_primary csr_pool ROBJ0 2026-02-13T02:14:05.639 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=csr_pool 2026-02-13T02:14:05.639 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=ROBJ0 2026-02-13T02:14:05.639 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map csr_pool ROBJ0 2026-02-13T02:14:05.639 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T02:14:05.797 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1079: TEST_corrupt_scrub_replicated: local primary=1 2026-02-13T02:14:05.798 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1082: TEST_corrupt_scrub_replicated: get_asok_path osd.0 2026-02-13T02:14:05.799 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-13T02:14:05.799 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-13T02:14:05.799 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:14:05.799 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:14:05.799 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:14:05.800 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-osd.0.asok 2026-02-13T02:14:05.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1082: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2026-02-13T02:14:05.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1082: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.35207/ceph-osd.0.asok config set osd_deep_scrub_update_digest_min_age 0 2026-02-13T02:14:05.856 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:14:05.856 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2026-02-13T02:14:05.856 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:14:05.862 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1084: TEST_corrupt_scrub_replicated: get_asok_path osd.1 2026-02-13T02:14:05.862 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-13T02:14:05.863 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-13T02:14:05.863 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:14:05.864 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:14:05.864 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:14:05.864 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-osd.1.asok 2026-02-13T02:14:05.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1084: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2026-02-13T02:14:05.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1084: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.35207/ceph-osd.1.asok config set osd_deep_scrub_update_digest_min_age 0 2026-02-13T02:14:05.917 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:14:05.917 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2026-02-13T02:14:05.917 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:14:05.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1086: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2026-02-13T02:14:05.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1932: pg_deep_scrub: local pgid=3.0 2026-02-13T02:14:05.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: pg_deep_scrub: wait_for_pg_clean 3.0 2026-02-13T02:14:05.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-13T02:14:05.924 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-13T02:14:05.924 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:14:05.925 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:14:05.925 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:14:05.925 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:14:05.925 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:14:06.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-13T02:14:06.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-13T02:14:06.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-13T02:14:06.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-13T02:14:06.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:14:06.214 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:14:06.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:14:06.375 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:14:06.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:14:06.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:14:06.375 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:14:06.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836488 2026-02-13T02:14:06.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836488 2026-02-13T02:14:06.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488' 2026-02-13T02:14:06.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:14:06.452 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:14:06.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705672 2026-02-13T02:14:06.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705672 2026-02-13T02:14:06.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-38654705672' 2026-02-13T02:14:06.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:14:06.525 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836488 2026-02-13T02:14:06.525 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:14:06.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:14:06.527 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836488 2026-02-13T02:14:06.527 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:14:06.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836488 2026-02-13T02:14:06.529 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836488 2026-02-13T02:14:06.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2026-02-13T02:14:06.529 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:14:06.692 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836488 2026-02-13T02:14:06.692 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:14:06.693 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705672 2026-02-13T02:14:06.693 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:14:06.694 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:14:06.695 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705672 2026-02-13T02:14:06.695 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:14:06.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705672 2026-02-13T02:14:06.697 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705672 2026-02-13T02:14:06.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705672' 2026-02-13T02:14:06.697 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:14:06.871 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705672 -lt 38654705672 2026-02-13T02:14:06.872 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-13T02:14:06.872 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-13T02:14:06.872 INFO:tasks.workunit.client.0.trial112.stdout:#---------- 3.0 loop 0 2026-02-13T02:14:06.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-13T02:14:06.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-13T02:14:06.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-13T02:14:06.873 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-13T02:14:06.873 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-13T02:14:06.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-13T02:14:06.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-13T02:14:06.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-13T02:14:06.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-13T02:14:06.948 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-13T02:14:06.948 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:14:06.948 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:14:06.948 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:14:06.948 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-13T02:14:07.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: local last_scrub=2026-02-13T02:13:51.267985+0000 2026-02-13T02:14:07.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: pg_deep_scrub: ceph pg deep-scrub 3.0 2026-02-13T02:14:07.262 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2026-02-13T02:14:07.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: pg_deep_scrub: wait_for_scrub 3.0 2026-02-13T02:13:51.267985+0000 last_deep_scrub_stamp 2026-02-13T02:14:07.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-13T02:14:07.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:13:51.267985+0000 2026-02-13T02:14:07.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_deep_scrub_stamp 2026-02-13T02:14:07.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:14:07.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:14:07.273 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-13T02:14:07.273 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:14:07.273 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:14:07.273 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:14:07.274 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-13T02:14:07.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:13:51.267985+0000 '>' 2026-02-13T02:13:51.267985+0000 2026-02-13T02:14:07.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:14:08.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:14:08.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:14:08.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-13T02:14:08.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:14:08.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:14:08.434 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:14:08.434 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-13T02:14:08.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:13:51.267985+0000 '>' 2026-02-13T02:13:51.267985+0000 2026-02-13T02:14:08.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:14:09.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:14:09.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:14:09.591 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-13T02:14:09.591 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:14:09.591 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:14:09.592 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:14:09.592 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-13T02:14:09.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:13:51.267985+0000 '>' 2026-02-13T02:13:51.267985+0000 2026-02-13T02:14:09.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:14:10.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:14:10.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:14:10.753 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-13T02:14:10.754 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:14:10.754 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:14:10.754 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:14:10.754 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-13T02:14:10.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:14:07.964003+0000 '>' 2026-02-13T02:13:51.267985+0000 2026-02-13T02:14:10.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:14:10.913 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: seq 1 19 2026-02-13T02:14:10.913 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-13T02:14:10.913 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ1 2026-02-13T02:14:10.914 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 1 % 2 2026-02-13T02:14:10.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=1 2026-02-13T02:14:10.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-13T02:14:10.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1097: TEST_corrupt_scrub_replicated: local payload=UVWXYZZZ 2026-02-13T02:14:10.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1098: TEST_corrupt_scrub_replicated: echo UVWXYZZZ 2026-02-13T02:14:10.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1099: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-13T02:14:10.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:14:10.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:14:10.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:14:10.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:14:10.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-13T02:14:10.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:14:10.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:14:10.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:14:10.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:14:10.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:14:10.918 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:14:10.918 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:14:10.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:14:10.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:14:10.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:14:11.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:14:11.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-13T02:14:11.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:14:11.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:14:11.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:14:11.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:14:11.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:14:11.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-13T02:14:11.423 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 3.0_head,#-5:00000000:::deep_scrub_3.0:head#, (61) No data available 2026-02-13T02:14:11.423 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 3.0_head,#-5:00000000:::scrub_3.0:head#, (61) No data available 2026-02-13T02:14:11.515 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-13T02:14:11.515 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:14:11.515 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:14:11.515 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:14:11.516 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:14:11.516 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:14:11.516 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:14:11.516 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:14:11.516 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:14:11.516 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:14:11.516 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:14:11.516 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:14:11.516 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:14:11.516 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:14:11.516 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:14:11.516 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:14:11.517 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:14:11.517 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:14:11.517 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:14:11.517 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:14:11.517 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:14:11.518 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:14:11.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:14:11.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:14:11.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:14:11.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:14:11.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:14:11.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:14:11.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:14:11.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:14:11.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:14:11.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:14:11.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:14:11.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:14:11.520 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:14:11.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:14:11.520 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:14:11.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:14:11.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:14:11.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:14:11.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:14:11.534 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:11.519+0000 7fb725345a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:14:11.535 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:11.519+0000 7fb725345a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:14:11.536 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:11.519+0000 7fb725345a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:14:11.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:14:11.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:14:11.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:14:11.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:14:11.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:14:11.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:14:11.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:14:11.685 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:14:11.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:14:11.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:14:11.847 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:14:11.887 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:11.871+0000 7fb725345a40 -1 Falling back to public interface 2026-02-13T02:14:12.430 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:12.415+0000 7fb725345a40 -1 osd.1 61 log_to_monitors true 2026-02-13T02:14:12.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:14:12.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:14:12.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:14:12.849 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:14:12.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:14:12.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:14:13.007 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 65 up_thru 65 down_at 62 last_clean_interval [9,61) [v2:127.0.0.1:6810/3053780495,v1:127.0.0.1:6811/3053780495] [v2:127.0.0.1:6812/3053780495,v1:127.0.0.1:6813/3053780495] exists,up cfe9e8e4-e309-485e-8178-7cb3d645195b 2026-02-13T02:14:13.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:14:13.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:14:13.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:14:13.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:14:13.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:14:13.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:14:13.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:14:13.007 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:14:13.008 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:14:13.008 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:14:13.009 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:14:13.009 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:14:13.009 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:14:13.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:14:13.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:14:13.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:14:13.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:14:13.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:14:13.115 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:14:13.271 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:14:13.271 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:14:13.271 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:14:13.271 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:14:13.272 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:14:13.343 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836491 2026-02-13T02:14:13.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836491 2026-02-13T02:14:13.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491' 2026-02-13T02:14:13.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:14:13.344 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:14:13.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=279172874242 2026-02-13T02:14:13.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 279172874242 2026-02-13T02:14:13.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-279172874242' 2026-02-13T02:14:13.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:14:13.418 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836491 2026-02-13T02:14:13.418 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:14:13.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:14:13.421 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836491 2026-02-13T02:14:13.421 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:14:13.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836491 2026-02-13T02:14:13.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2026-02-13T02:14:13.423 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 21474836491 2026-02-13T02:14:13.423 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:14:13.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836490 -lt 21474836491 2026-02-13T02:14:13.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:14:14.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:14:14.580 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:14:14.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836491 2026-02-13T02:14:14.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:14:14.744 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-279172874242 2026-02-13T02:14:14.744 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:14:14.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:14:14.746 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-279172874242 2026-02-13T02:14:14.746 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:14:14.748 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=279172874242 2026-02-13T02:14:14.748 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 279172874242' 2026-02-13T02:14:14.748 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 279172874242 2026-02-13T02:14:14.749 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:14:14.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 279172874242 -lt 279172874242 2026-02-13T02:14:14.922 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:14:14.922 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:14:14.923 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:14:15.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:14:15.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:14:15.118 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:14:15.119 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:14:15.119 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:14:15.119 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:14:15.119 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:14:15.119 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:14:15.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:14:15.280 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:14:15.280 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:14:15.281 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:14:15.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:14:15.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:14:15.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:14:15.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-13T02:14:15.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ2 2026-02-13T02:14:15.475 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 2 % 2 2026-02-13T02:14:15.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=0 2026-02-13T02:14:15.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-13T02:14:15.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1104: TEST_corrupt_scrub_replicated: local payload=UVWXYZ 2026-02-13T02:14:15.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1105: TEST_corrupt_scrub_replicated: echo UVWXYZ 2026-02-13T02:14:15.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1106: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-13T02:14:15.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:14:15.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:14:15.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T02:14:15.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:14:15.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-13T02:14:15.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:14:15.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:14:15.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T02:14:15.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:14:15.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T02:14:15.479 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:14:15.479 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:14:15.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:14:15.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:14:15.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:14:15.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:14:15.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-13T02:14:15.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:14:15.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:14:15.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:14:15.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:14:15.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:14:15.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-13T02:14:16.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-13T02:14:16.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:14:16.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:14:16.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:14:16.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:14:16.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:14:16.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:14:16.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:14:16.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:14:16.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:14:16.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:14:16.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:14:16.082 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:14:16.082 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:14:16.082 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:14:16.082 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:14:16.082 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:14:16.082 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:14:16.082 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:14:16.082 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:14:16.082 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:14:16.082 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:14:16.082 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:14:16.082 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:14:16.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:14:16.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:14:16.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:14:16.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:14:16.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:14:16.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:14:16.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:14:16.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:14:16.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:14:16.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:14:16.084 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T02:14:16.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:14:16.085 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T02:14:16.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:14:16.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:14:16.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:14:16.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:14:16.101 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:16.083+0000 7f9cba1dfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:14:16.102 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:16.087+0000 7f9cba1dfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:14:16.103 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:16.087+0000 7f9cba1dfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:14:16.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:14:16.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:14:16.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:14:16.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:14:16.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:14:16.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:14:16.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:14:16.248 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:14:16.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:14:16.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:14:16.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:14:16.527 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:16.511+0000 7f9cba1dfa40 -1 Falling back to public interface 2026-02-13T02:14:17.082 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:17.067+0000 7f9cba1dfa40 -1 osd.0 68 log_to_monitors true 2026-02-13T02:14:17.410 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:14:17.410 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:14:17.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:14:17.411 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:14:17.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:14:17.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:14:17.576 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 71 up_thru 71 down_at 68 last_clean_interval [5,68) [v2:127.0.0.1:6800/3732460796,v1:127.0.0.1:6801/3732460796] [v2:127.0.0.1:6802/3732460796,v1:127.0.0.1:6803/3732460796] exists,up 926506f0-916f-45ab-94cb-3aa841154594 2026-02-13T02:14:17.576 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:14:17.576 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:14:17.576 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:14:17.576 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:14:17.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:14:17.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:14:17.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:14:17.577 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:14:17.578 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:14:17.578 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:14:17.578 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:14:17.578 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:14:17.578 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:14:17.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:14:17.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:14:17.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:14:17.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:14:17.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:14:17.688 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:14:17.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:14:17.846 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:14:17.847 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:14:17.847 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:14:17.847 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:14:17.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=304942678018 2026-02-13T02:14:17.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 304942678018 2026-02-13T02:14:17.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-304942678018' 2026-02-13T02:14:17.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:14:17.924 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:14:17.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=279172874244 2026-02-13T02:14:17.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 279172874244 2026-02-13T02:14:17.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-304942678018 1-279172874244' 2026-02-13T02:14:17.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:14:17.997 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-304942678018 2026-02-13T02:14:17.997 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:14:17.998 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:14:17.999 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-304942678018 2026-02-13T02:14:17.999 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:14:18.001 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=304942678018 2026-02-13T02:14:18.001 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 304942678018' 2026-02-13T02:14:18.001 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 304942678018 2026-02-13T02:14:18.001 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:14:18.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 304942678018 2026-02-13T02:14:18.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:14:19.175 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:14:19.176 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:14:19.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 304942678018 -lt 304942678018 2026-02-13T02:14:19.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:14:19.338 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-279172874244 2026-02-13T02:14:19.338 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:14:19.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:14:19.341 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-279172874244 2026-02-13T02:14:19.341 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:14:19.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=279172874244 2026-02-13T02:14:19.343 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 279172874244' 2026-02-13T02:14:19.343 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 279172874244 2026-02-13T02:14:19.343 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:14:19.509 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 279172874244 -lt 279172874244 2026-02-13T02:14:19.509 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:14:19.510 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:14:19.510 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:14:19.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:14:19.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:14:19.709 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:14:19.709 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:14:19.709 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:14:19.710 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:14:19.710 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:14:19.710 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:14:19.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:14:19.869 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:14:19.869 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:14:19.870 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:14:20.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:14:20.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:14:20.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:14:20.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-13T02:14:20.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ3 2026-02-13T02:14:20.070 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 3 % 2 2026-02-13T02:14:20.072 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=1 2026-02-13T02:14:20.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-13T02:14:20.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1111: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ3 remove 2026-02-13T02:14:20.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:14:20.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:14:20.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:14:20.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:14:20.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ3 remove 2026-02-13T02:14:20.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:14:20.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:14:20.074 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:14:20.074 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:14:20.074 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:14:20.074 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:14:20.074 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:14:20.074 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:14:20.074 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:14:20.074 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:14:20.382 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:14:20.382 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ3 remove 2026-02-13T02:14:20.382 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:14:20.382 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:14:20.382 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:14:20.382 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:14:20.383 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:14:20.383 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ3 remove 2026-02-13T02:14:20.554 INFO:tasks.workunit.client.0.trial112.stdout:remove #3:f2a5b2a4:::ROBJ3:head# 2026-02-13T02:14:20.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-13T02:14:20.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:14:20.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:14:20.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:14:20.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:14:20.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:14:20.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:14:20.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:14:20.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:14:20.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:14:20.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:14:20.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:14:20.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:14:20.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:14:20.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:14:20.657 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:14:20.657 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:14:20.657 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:14:20.657 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:14:20.657 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:14:20.657 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:14:20.657 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:14:20.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:14:20.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:14:20.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:14:20.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:14:20.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:14:20.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:14:20.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:14:20.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:14:20.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:14:20.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:14:20.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:14:20.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:14:20.660 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:14:20.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:14:20.660 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:14:20.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:14:20.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:14:20.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:14:20.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:14:20.676 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:20.659+0000 7f7717adda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:14:20.678 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:20.663+0000 7f7717adda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:14:20.679 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:20.663+0000 7f7717adda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:14:20.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:14:20.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:14:20.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:14:20.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:14:20.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:14:20.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:14:20.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:14:20.825 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:14:20.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:14:20.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:14:20.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:14:21.027 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:21.011+0000 7f7717adda40 -1 Falling back to public interface 2026-02-13T02:14:21.572 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:21.555+0000 7f7717adda40 -1 osd.1 74 log_to_monitors true 2026-02-13T02:14:21.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:14:21.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:14:21.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:14:21.987 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:14:21.988 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:14:21.988 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:14:22.144 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 78 up_thru 78 down_at 75 last_clean_interval [65,74) [v2:127.0.0.1:6810/2300919913,v1:127.0.0.1:6811/2300919913] [v2:127.0.0.1:6812/2300919913,v1:127.0.0.1:6813/2300919913] exists,up cfe9e8e4-e309-485e-8178-7cb3d645195b 2026-02-13T02:14:22.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:14:22.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:14:22.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:14:22.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:14:22.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:14:22.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:14:22.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:14:22.146 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:14:22.146 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:14:22.146 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:14:22.146 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:14:22.147 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:14:22.147 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:14:22.267 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:14:22.267 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:14:22.267 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:14:22.267 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:14:22.267 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:14:22.267 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:14:22.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:14:22.428 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:14:22.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:14:22.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:14:22.429 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:14:22.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=304942678020 2026-02-13T02:14:22.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 304942678020 2026-02-13T02:14:22.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-304942678020' 2026-02-13T02:14:22.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:14:22.502 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:14:22.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=335007449090 2026-02-13T02:14:22.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 335007449090 2026-02-13T02:14:22.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-304942678020 1-335007449090' 2026-02-13T02:14:22.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:14:22.578 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-304942678020 2026-02-13T02:14:22.578 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:14:22.580 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:14:22.580 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-304942678020 2026-02-13T02:14:22.581 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:14:22.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=304942678020 2026-02-13T02:14:22.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 304942678020' 2026-02-13T02:14:22.583 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 304942678020 2026-02-13T02:14:22.583 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:14:22.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 304942678020 -lt 304942678020 2026-02-13T02:14:22.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:14:22.742 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-335007449090 2026-02-13T02:14:22.742 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:14:22.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:14:22.744 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-335007449090 2026-02-13T02:14:22.744 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:14:22.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=335007449090 2026-02-13T02:14:22.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 335007449090' 2026-02-13T02:14:22.746 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 335007449090 2026-02-13T02:14:22.746 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:14:22.910 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 335007449090 2026-02-13T02:14:22.910 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:14:23.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:14:23.912 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:14:24.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 335007449090 2026-02-13T02:14:24.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:14:25.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:14:25.082 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:14:25.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 335007449090 -lt 335007449090 2026-02-13T02:14:25.241 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:14:25.242 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:14:25.242 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:14:25.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:14:25.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:14:25.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:14:25.434 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:14:25.434 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:14:25.434 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:14:25.434 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:14:25.434 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:14:25.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:14:25.594 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:14:25.594 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:14:25.594 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:14:25.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:14:25.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:14:25.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:14:25.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-13T02:14:25.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ4 2026-02-13T02:14:25.794 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 4 % 2 2026-02-13T02:14:25.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=0 2026-02-13T02:14:25.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-13T02:14:25.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1116: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2026-02-13T02:14:25.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:14:25.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:14:25.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T02:14:25.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:14:25.797 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2026-02-13T02:14:25.797 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:14:25.797 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:14:25.797 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T02:14:25.797 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:14:25.797 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T02:14:25.797 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:14:25.797 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:14:25.797 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:14:25.797 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:14:25.797 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:14:26.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:14:26.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2026-02-13T02:14:26.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:14:26.107 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:14:26.107 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:14:26.107 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:14:26.107 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:14:26.107 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2026-02-13T02:14:26.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-13T02:14:26.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:14:26.361 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:14:26.361 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:14:26.361 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:14:26.361 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:14:26.361 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:14:26.361 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:14:26.361 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:14:26.361 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:14:26.361 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:14:26.361 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:14:26.361 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:14:26.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:14:26.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:14:26.362 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:14:26.362 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:14:26.362 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:14:26.362 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:14:26.362 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:14:26.362 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:14:26.362 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:14:26.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:14:26.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:14:26.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:14:26.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:14:26.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:14:26.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:14:26.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:14:26.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:14:26.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:14:26.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:14:26.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:14:26.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:14:26.364 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T02:14:26.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:14:26.365 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T02:14:26.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:14:26.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:14:26.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:14:26.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:14:26.379 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:26.366+0000 7ff5fa1aea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:14:26.380 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:26.366+0000 7ff5fa1aea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:14:26.381 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:26.366+0000 7ff5fa1aea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:14:26.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:14:26.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:14:26.527 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:14:26.527 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:14:26.527 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:14:26.527 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:14:26.527 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:14:26.527 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:14:26.527 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:14:26.527 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:14:26.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:14:26.795 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:26.782+0000 7ff5fa1aea40 -1 Falling back to public interface 2026-02-13T02:14:27.340 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:27.326+0000 7ff5fa1aea40 -1 osd.0 79 log_to_monitors true 2026-02-13T02:14:27.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:14:27.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:14:27.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:14:27.686 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:14:27.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:14:27.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:14:27.849 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 83 up_thru 83 down_at 80 last_clean_interval [71,79) [v2:127.0.0.1:6800/2394030687,v1:127.0.0.1:6801/2394030687] [v2:127.0.0.1:6802/2394030687,v1:127.0.0.1:6803/2394030687] exists,up 926506f0-916f-45ab-94cb-3aa841154594 2026-02-13T02:14:27.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:14:27.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:14:27.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:14:27.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:14:27.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:14:27.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:14:27.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:14:27.850 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:14:27.851 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:14:27.851 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:14:27.851 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:14:27.851 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:14:27.851 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:14:27.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:14:27.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:14:27.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:14:27.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:14:27.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:14:27.962 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:14:28.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:14:28.118 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:14:28.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:14:28.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:14:28.118 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:14:28.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=356482285570 2026-02-13T02:14:28.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 356482285570 2026-02-13T02:14:28.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-356482285570' 2026-02-13T02:14:28.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:14:28.193 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:14:28.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=335007449092 2026-02-13T02:14:28.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 335007449092 2026-02-13T02:14:28.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-356482285570 1-335007449092' 2026-02-13T02:14:28.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:14:28.265 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-356482285570 2026-02-13T02:14:28.265 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:14:28.266 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:14:28.267 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-356482285570 2026-02-13T02:14:28.267 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:14:28.269 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=356482285570 2026-02-13T02:14:28.269 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 356482285570' 2026-02-13T02:14:28.269 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 356482285570 2026-02-13T02:14:28.270 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:14:28.442 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 304942678020 -lt 356482285570 2026-02-13T02:14:28.442 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:14:29.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:14:29.444 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:14:29.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 356482285570 -lt 356482285570 2026-02-13T02:14:29.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:14:29.605 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-335007449092 2026-02-13T02:14:29.605 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:14:29.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:14:29.607 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-335007449092 2026-02-13T02:14:29.607 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:14:29.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=335007449092 2026-02-13T02:14:29.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 335007449092' 2026-02-13T02:14:29.609 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 335007449092 2026-02-13T02:14:29.610 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:14:29.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 335007449092 -lt 335007449092 2026-02-13T02:14:29.782 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:14:29.783 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:14:29.783 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:14:29.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:14:29.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:14:29.985 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:14:29.985 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:14:29.985 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:14:29.985 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:14:29.985 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:14:29.985 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:14:30.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:14:30.150 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:14:30.151 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:14:30.151 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:14:30.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:14:30.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:14:30.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:14:30.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-13T02:14:30.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ5 2026-02-13T02:14:30.352 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 5 % 2 2026-02-13T02:14:30.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=1 2026-02-13T02:14:30.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-13T02:14:30.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1121: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2026-02-13T02:14:30.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:14:30.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:14:30.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:14:30.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:14:30.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2026-02-13T02:14:30.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:14:30.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:14:30.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:14:30.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:14:30.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:14:30.355 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:14:30.355 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:14:30.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:14:30.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:14:30.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:14:30.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:14:30.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2026-02-13T02:14:30.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:14:30.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:14:30.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:14:30.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:14:30.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:14:30.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ5 rm-omap key-ROBJ5 2026-02-13T02:14:30.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-13T02:14:30.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:14:30.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:14:30.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:14:30.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:14:30.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:14:30.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:14:30.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:14:30.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:14:30.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:14:30.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:14:30.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:14:30.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:14:30.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:14:30.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:14:30.942 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:14:30.942 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:14:30.942 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:14:30.942 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:14:30.943 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:14:30.943 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:14:30.943 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:14:30.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:14:30.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:14:30.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:14:30.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:14:30.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:14:30.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:14:30.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:14:30.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:14:30.944 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:14:30.944 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:14:30.944 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:14:30.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:14:30.945 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:14:30.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:14:30.945 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:14:30.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:14:30.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:14:30.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:14:30.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:14:30.964 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:30.950+0000 7ffac118aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:14:30.966 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:30.954+0000 7ffac118aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:14:30.967 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:30.954+0000 7ffac118aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:14:31.099 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:14:31.099 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:14:31.099 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:14:31.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:14:31.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:14:31.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:14:31.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:14:31.100 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:14:31.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:14:31.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:14:31.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:14:31.351 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:31.338+0000 7ffac118aa40 -1 Falling back to public interface 2026-02-13T02:14:31.992 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:31.978+0000 7ffac118aa40 -1 osd.1 84 log_to_monitors true 2026-02-13T02:14:32.267 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:14:32.267 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:14:32.267 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:14:32.267 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:14:32.267 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:14:32.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:14:32.429 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 88 up_thru 88 down_at 85 last_clean_interval [78,84) [v2:127.0.0.1:6810/356910078,v1:127.0.0.1:6811/356910078] [v2:127.0.0.1:6812/356910078,v1:127.0.0.1:6813/356910078] exists,up cfe9e8e4-e309-485e-8178-7cb3d645195b 2026-02-13T02:14:32.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:14:32.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:14:32.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:14:32.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:14:32.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:14:32.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:14:32.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:14:32.430 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:14:32.431 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:14:32.431 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:14:32.431 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:14:32.431 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:14:32.431 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:14:32.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:14:32.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:14:32.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:14:32.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:14:32.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:14:32.546 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:14:32.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:14:32.712 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:14:32.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:14:32.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:14:32.713 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:14:32.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=356482285572 2026-02-13T02:14:32.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 356482285572 2026-02-13T02:14:32.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-356482285572' 2026-02-13T02:14:32.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:14:32.785 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:14:32.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=377957122050 2026-02-13T02:14:32.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 377957122050 2026-02-13T02:14:32.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-356482285572 1-377957122050' 2026-02-13T02:14:32.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:14:32.857 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-356482285572 2026-02-13T02:14:32.857 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:14:32.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:14:32.859 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-356482285572 2026-02-13T02:14:32.859 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:14:32.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=356482285572 2026-02-13T02:14:32.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 356482285572' 2026-02-13T02:14:32.861 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 356482285572 2026-02-13T02:14:32.862 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:14:33.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 356482285571 -lt 356482285572 2026-02-13T02:14:33.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:14:34.037 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:14:34.038 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:14:34.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 356482285571 -lt 356482285572 2026-02-13T02:14:34.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:14:35.201 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:14:35.202 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:14:35.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 356482285572 -lt 356482285572 2026-02-13T02:14:35.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:14:35.366 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-377957122050 2026-02-13T02:14:35.366 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:14:35.368 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:14:35.368 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-377957122050 2026-02-13T02:14:35.369 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:14:35.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=377957122050 2026-02-13T02:14:35.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 377957122050' 2026-02-13T02:14:35.371 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 377957122050 2026-02-13T02:14:35.371 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:14:35.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 377957122050 -lt 377957122050 2026-02-13T02:14:35.533 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:14:35.534 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:14:35.534 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:14:35.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:14:35.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:14:35.735 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:14:35.735 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:14:35.735 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:14:35.735 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:14:35.735 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:14:35.736 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:14:35.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:14:35.898 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:14:35.898 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:14:35.898 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:14:36.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:14:36.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:14:36.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:14:36.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-13T02:14:36.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ6 2026-02-13T02:14:36.088 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 6 % 2 2026-02-13T02:14:36.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=0 2026-02-13T02:14:36.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-13T02:14:36.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1126: TEST_corrupt_scrub_replicated: echo extra 2026-02-13T02:14:36.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1127: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2026-02-13T02:14:36.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:14:36.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:14:36.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T02:14:36.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:14:36.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2026-02-13T02:14:36.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:14:36.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:14:36.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T02:14:36.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:14:36.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T02:14:36.092 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:14:36.092 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:14:36.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:14:36.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:14:36.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:14:36.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:14:36.401 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2026-02-13T02:14:36.401 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:14:36.401 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:14:36.401 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:14:36.401 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:14:36.401 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:14:36.401 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2026-02-13T02:14:36.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-13T02:14:36.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:14:36.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:14:36.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:14:36.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:14:36.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:14:36.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:14:36.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:14:36.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:14:36.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:14:36.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:14:36.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:14:36.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:14:36.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:14:36.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:14:36.673 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:14:36.673 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:14:36.673 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:14:36.673 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:14:36.673 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:14:36.673 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:14:36.675 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:14:36.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:14:36.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:14:36.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:14:36.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:14:36.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:14:36.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:14:36.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:14:36.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:14:36.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:14:36.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:14:36.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:14:36.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:14:36.676 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T02:14:36.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:14:36.677 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T02:14:36.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:14:36.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:14:36.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:14:36.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:14:36.690 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:36.678+0000 7f00b202fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:14:36.691 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:36.678+0000 7f00b202fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:14:36.692 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:36.678+0000 7f00b202fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:14:36.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:14:36.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:14:36.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:14:36.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:14:36.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:14:36.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:14:36.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:14:36.841 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:14:36.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:14:36.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:14:36.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:14:37.067 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:37.054+0000 7f00b202fa40 -1 Falling back to public interface 2026-02-13T02:14:37.723 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:37.710+0000 7f00b202fa40 -1 osd.0 89 log_to_monitors true 2026-02-13T02:14:38.001 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:14:38.001 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:14:38.001 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:14:38.001 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:14:38.001 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:14:38.001 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:14:38.156 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 93 up_thru 93 down_at 90 last_clean_interval [83,89) [v2:127.0.0.1:6800/888136785,v1:127.0.0.1:6801/888136785] [v2:127.0.0.1:6802/888136785,v1:127.0.0.1:6803/888136785] exists,up 926506f0-916f-45ab-94cb-3aa841154594 2026-02-13T02:14:38.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:14:38.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:14:38.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:14:38.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:14:38.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:14:38.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:14:38.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:14:38.157 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:14:38.158 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:14:38.158 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:14:38.158 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:14:38.158 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:14:38.158 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:14:38.277 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:14:38.277 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:14:38.277 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:14:38.277 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:14:38.277 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:14:38.277 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:14:38.441 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:14:38.441 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:14:38.441 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:14:38.441 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:14:38.441 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:14:38.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=399431958530 2026-02-13T02:14:38.515 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 399431958530 2026-02-13T02:14:38.515 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-399431958530' 2026-02-13T02:14:38.515 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:14:38.515 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:14:38.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=377957122052 2026-02-13T02:14:38.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 377957122052 2026-02-13T02:14:38.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-399431958530 1-377957122052' 2026-02-13T02:14:38.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:14:38.589 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-399431958530 2026-02-13T02:14:38.589 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:14:38.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:14:38.591 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-399431958530 2026-02-13T02:14:38.591 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:14:38.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=399431958530 2026-02-13T02:14:38.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 399431958530' 2026-02-13T02:14:38.593 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 399431958530 2026-02-13T02:14:38.593 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:14:38.766 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 399431958530 -lt 399431958530 2026-02-13T02:14:38.766 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:14:38.767 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-377957122052 2026-02-13T02:14:38.767 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:14:38.768 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:14:38.769 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-377957122052 2026-02-13T02:14:38.769 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:14:38.771 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=377957122052 2026-02-13T02:14:38.771 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 377957122052' 2026-02-13T02:14:38.771 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 377957122052 2026-02-13T02:14:38.771 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:14:38.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 377957122051 -lt 377957122052 2026-02-13T02:14:38.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:14:39.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:14:39.932 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:14:40.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 377957122051 -lt 377957122052 2026-02-13T02:14:40.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:14:41.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:14:41.098 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:14:41.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 377957122052 -lt 377957122052 2026-02-13T02:14:41.259 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:14:41.260 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:14:41.260 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:14:41.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:14:41.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:14:41.450 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:14:41.450 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:14:41.450 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:14:41.450 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:14:41.450 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:14:41.450 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:14:41.613 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:14:41.614 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:14:41.615 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:14:41.615 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:14:41.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:14:41.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:14:41.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:14:41.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1128: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/extra-val 2026-02-13T02:14:41.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-13T02:14:41.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ7 2026-02-13T02:14:41.814 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 7 % 2 2026-02-13T02:14:41.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=1 2026-02-13T02:14:41.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-13T02:14:41.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1133: TEST_corrupt_scrub_replicated: echo -n newheader 2026-02-13T02:14:41.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1134: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2026-02-13T02:14:41.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:14:41.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:14:41.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:14:41.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:14:41.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2026-02-13T02:14:41.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:14:41.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:14:41.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:14:41.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:14:41.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:14:41.818 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:14:41.818 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:14:41.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:14:41.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:14:41.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:14:42.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:14:42.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2026-02-13T02:14:42.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:14:42.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:14:42.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:14:42.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:14:42.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:14:42.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2026-02-13T02:14:42.383 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-13T02:14:42.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:14:42.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:14:42.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:14:42.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:14:42.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:14:42.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:14:42.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:14:42.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:14:42.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:14:42.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:14:42.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:14:42.385 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:14:42.385 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:14:42.385 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:14:42.385 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:14:42.385 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:14:42.385 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:14:42.385 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:14:42.385 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:14:42.385 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:14:42.385 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:14:42.385 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:14:42.385 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:14:42.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:14:42.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:14:42.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:14:42.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:14:42.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:14:42.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:14:42.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:14:42.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:14:42.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:14:42.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:14:42.387 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:14:42.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:14:42.388 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:14:42.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:14:42.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:14:42.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:14:42.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:14:42.402 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:42.390+0000 7f0fc0f5da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:14:42.404 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:42.390+0000 7f0fc0f5da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:14:42.405 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:42.390+0000 7f0fc0f5da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:14:42.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:14:42.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:14:42.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:14:42.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:14:42.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:14:42.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:14:42.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:14:42.552 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:14:42.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:14:42.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:14:42.707 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:14:42.759 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:42.746+0000 7f0fc0f5da40 -1 Falling back to public interface 2026-02-13T02:14:43.303 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:43.290+0000 7f0fc0f5da40 -1 osd.1 94 log_to_monitors true 2026-02-13T02:14:43.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:14:43.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:14:43.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:14:43.709 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:14:43.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:14:43.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:14:43.869 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 98 up_thru 98 down_at 95 last_clean_interval [88,94) [v2:127.0.0.1:6810/417805404,v1:127.0.0.1:6811/417805404] [v2:127.0.0.1:6812/417805404,v1:127.0.0.1:6813/417805404] exists,up cfe9e8e4-e309-485e-8178-7cb3d645195b 2026-02-13T02:14:43.869 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:14:43.869 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:14:43.869 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:14:43.869 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:14:43.869 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:14:43.869 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:14:43.870 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:14:43.871 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:14:43.871 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:14:43.871 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:14:43.871 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:14:43.871 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:14:43.871 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:14:43.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:14:43.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:14:43.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:14:43.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:14:43.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:14:43.991 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:14:44.152 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:14:44.152 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:14:44.152 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:14:44.152 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:14:44.153 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:14:44.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=399431958532 2026-02-13T02:14:44.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 399431958532 2026-02-13T02:14:44.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-399431958532' 2026-02-13T02:14:44.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:14:44.226 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:14:44.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=420906795010 2026-02-13T02:14:44.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 420906795010 2026-02-13T02:14:44.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-399431958532 1-420906795010' 2026-02-13T02:14:44.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:14:44.299 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-399431958532 2026-02-13T02:14:44.299 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:14:44.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:14:44.302 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-399431958532 2026-02-13T02:14:44.302 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:14:44.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=399431958532 2026-02-13T02:14:44.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 399431958532' 2026-02-13T02:14:44.303 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 399431958532 2026-02-13T02:14:44.304 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:14:44.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 399431958530 -lt 399431958532 2026-02-13T02:14:44.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:14:45.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:14:45.473 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:14:45.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 399431958532 -lt 399431958532 2026-02-13T02:14:45.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:14:45.637 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-420906795010 2026-02-13T02:14:45.637 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:14:45.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:14:45.639 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-420906795010 2026-02-13T02:14:45.639 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:14:45.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=420906795010 2026-02-13T02:14:45.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 420906795010' 2026-02-13T02:14:45.641 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 420906795010 2026-02-13T02:14:45.642 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:14:45.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 420906795010 -lt 420906795010 2026-02-13T02:14:45.810 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:14:45.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:14:45.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:14:46.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:14:46.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:14:46.014 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:14:46.014 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:14:46.014 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:14:46.014 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:14:46.014 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:14:46.014 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:14:46.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:14:46.180 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:14:46.181 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:14:46.181 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:14:46.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:14:46.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:14:46.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:14:46.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1135: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/hdr 2026-02-13T02:14:46.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-13T02:14:46.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ8 2026-02-13T02:14:46.380 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 8 % 2 2026-02-13T02:14:46.381 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=0 2026-02-13T02:14:46.381 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-13T02:14:46.381 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1139: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key1-ROBJ8 val1-ROBJ8 2026-02-13T02:14:46.401 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1140: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key2-ROBJ8 val2-ROBJ8 2026-02-13T02:14:46.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1143: TEST_corrupt_scrub_replicated: echo -n bad-val 2026-02-13T02:14:46.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1144: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2026-02-13T02:14:46.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:14:46.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:14:46.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T02:14:46.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:14:46.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2026-02-13T02:14:46.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:14:46.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:14:46.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T02:14:46.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:14:46.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T02:14:46.429 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:14:46.429 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:14:46.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:14:46.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:14:46.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:14:46.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:14:46.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2026-02-13T02:14:46.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:14:46.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:14:46.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:14:46.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:14:46.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:14:46.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2026-02-13T02:14:47.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-13T02:14:47.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:14:47.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:14:47.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:14:47.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:14:47.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:14:47.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:14:47.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:14:47.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:14:47.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:14:47.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:14:47.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:14:47.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:14:47.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:14:47.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:14:47.010 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:14:47.010 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:14:47.010 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:14:47.010 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:14:47.011 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:14:47.011 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:14:47.011 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:14:47.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:14:47.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:14:47.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:14:47.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:14:47.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:14:47.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:14:47.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:14:47.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:14:47.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:14:47.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:14:47.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:14:47.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:14:47.013 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T02:14:47.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:14:47.013 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T02:14:47.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:14:47.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:14:47.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:14:47.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:14:47.029 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:47.014+0000 7f204e178a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:14:47.030 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:47.018+0000 7f204e178a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:14:47.031 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:47.018+0000 7f204e178a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:14:47.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:14:47.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:14:47.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:14:47.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:14:47.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:14:47.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:14:47.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:14:47.174 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:14:47.175 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:14:47.175 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:14:47.332 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:14:47.420 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:47.406+0000 7f204e178a40 -1 Falling back to public interface 2026-02-13T02:14:48.057 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:48.042+0000 7f204e178a40 -1 osd.0 99 log_to_monitors true 2026-02-13T02:14:48.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:14:48.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:14:48.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:14:48.335 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:14:48.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:14:48.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:14:48.493 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 103 up_thru 103 down_at 100 last_clean_interval [93,99) [v2:127.0.0.1:6800/2899678520,v1:127.0.0.1:6801/2899678520] [v2:127.0.0.1:6802/2899678520,v1:127.0.0.1:6803/2899678520] exists,up 926506f0-916f-45ab-94cb-3aa841154594 2026-02-13T02:14:48.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:14:48.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:14:48.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:14:48.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:14:48.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:14:48.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:14:48.494 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:14:48.494 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:14:48.495 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:14:48.495 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:14:48.495 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:14:48.495 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:14:48.495 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:14:48.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:14:48.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:14:48.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:14:48.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:14:48.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:14:48.612 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:14:48.770 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:14:48.770 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:14:48.770 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:14:48.770 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:14:48.771 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:14:48.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=442381631490 2026-02-13T02:14:48.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 442381631490 2026-02-13T02:14:48.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-442381631490' 2026-02-13T02:14:48.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:14:48.844 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:14:48.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=420906795012 2026-02-13T02:14:48.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 420906795012 2026-02-13T02:14:48.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-442381631490 1-420906795012' 2026-02-13T02:14:48.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:14:48.917 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-442381631490 2026-02-13T02:14:48.917 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:14:48.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:14:48.919 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-442381631490 2026-02-13T02:14:48.919 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:14:48.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=442381631490 2026-02-13T02:14:48.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 442381631490' 2026-02-13T02:14:48.921 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 442381631490 2026-02-13T02:14:48.921 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:14:49.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 442381631490 2026-02-13T02:14:49.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:14:50.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:14:50.092 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:14:50.253 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 442381631490 2026-02-13T02:14:50.254 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:14:51.255 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:14:51.255 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:14:51.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 442381631490 -lt 442381631490 2026-02-13T02:14:51.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:14:51.421 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-420906795012 2026-02-13T02:14:51.421 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:14:51.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:14:51.423 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-420906795012 2026-02-13T02:14:51.423 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:14:51.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=420906795012 2026-02-13T02:14:51.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 420906795012' 2026-02-13T02:14:51.425 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 420906795012 2026-02-13T02:14:51.426 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:14:51.599 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 420906795012 -lt 420906795012 2026-02-13T02:14:51.600 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:14:51.600 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:14:51.600 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:14:51.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:14:51.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:14:51.797 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:14:51.797 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:14:51.797 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:14:51.797 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:14:51.797 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:14:51.797 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:14:51.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:14:51.952 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:14:51.953 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:14:51.953 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:14:52.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:14:52.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:14:52.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:14:52.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1145: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2026-02-13T02:14:52.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:14:52.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:14:52.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:14:52.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:14:52.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2026-02-13T02:14:52.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:14:52.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:14:52.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:14:52.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:14:52.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:14:52.148 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:14:52.148 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:14:52.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:14:52.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:14:52.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:14:52.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:14:52.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2026-02-13T02:14:52.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:14:52.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:14:52.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:14:52.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:14:52.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:14:52.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 rm-attr _key2-ROBJ8 2026-02-13T02:14:52.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-13T02:14:52.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:14:52.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:14:52.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:14:52.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:14:52.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:14:52.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:14:52.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:14:52.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:14:52.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:14:52.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:14:52.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:14:52.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:14:52.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:14:52.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:14:52.741 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:14:52.741 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:14:52.741 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:14:52.741 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:14:52.742 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:14:52.742 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:14:52.743 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:14:52.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:14:52.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:14:52.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:14:52.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:14:52.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:14:52.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:14:52.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:14:52.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:14:52.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:14:52.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:14:52.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:14:52.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:14:52.745 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:14:52.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:14:52.745 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:14:52.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:14:52.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:14:52.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:14:52.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:14:52.761 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:52.746+0000 7f5b3e748a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:14:52.762 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:52.750+0000 7f5b3e748a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:14:52.763 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:52.750+0000 7f5b3e748a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:14:52.908 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:14:52.908 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:14:52.908 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:14:52.908 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:14:52.908 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:14:52.908 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:14:52.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:14:52.909 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:14:52.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:14:52.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:14:53.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:14:53.151 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:53.138+0000 7f5b3e748a40 -1 Falling back to public interface 2026-02-13T02:14:53.713 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:53.698+0000 7f5b3e748a40 -1 osd.1 105 log_to_monitors true 2026-02-13T02:14:54.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:14:54.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:14:54.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:14:54.070 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:14:54.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:14:54.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:14:54.230 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 109 up_thru 109 down_at 106 last_clean_interval [98,105) [v2:127.0.0.1:6810/258354308,v1:127.0.0.1:6811/258354308] [v2:127.0.0.1:6812/258354308,v1:127.0.0.1:6813/258354308] exists,up cfe9e8e4-e309-485e-8178-7cb3d645195b 2026-02-13T02:14:54.230 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:14:54.230 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:14:54.230 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:14:54.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:14:54.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:14:54.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:14:54.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:14:54.231 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:14:54.232 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:14:54.232 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:14:54.232 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:14:54.232 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:14:54.232 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:14:54.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:14:54.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:14:54.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:14:54.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:14:54.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:14:54.342 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:14:54.508 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:14:54.508 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:14:54.509 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:14:54.509 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:14:54.509 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:14:54.581 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=442381631492 2026-02-13T02:14:54.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 442381631492 2026-02-13T02:14:54.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-442381631492' 2026-02-13T02:14:54.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:14:54.582 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:14:54.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=468151435266 2026-02-13T02:14:54.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 468151435266 2026-02-13T02:14:54.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-442381631492 1-468151435266' 2026-02-13T02:14:54.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:14:54.655 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-442381631492 2026-02-13T02:14:54.655 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:14:54.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:14:54.657 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-442381631492 2026-02-13T02:14:54.657 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:14:54.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=442381631492 2026-02-13T02:14:54.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 442381631492' 2026-02-13T02:14:54.659 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 442381631492 2026-02-13T02:14:54.659 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:14:54.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 442381631491 -lt 442381631492 2026-02-13T02:14:54.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:14:55.816 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:14:55.817 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:14:55.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 442381631491 -lt 442381631492 2026-02-13T02:14:55.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:14:56.980 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:14:56.981 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:14:57.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 442381631492 -lt 442381631492 2026-02-13T02:14:57.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:14:57.141 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-468151435266 2026-02-13T02:14:57.141 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:14:57.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:14:57.144 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-468151435266 2026-02-13T02:14:57.144 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:14:57.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=468151435266 2026-02-13T02:14:57.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 468151435266' 2026-02-13T02:14:57.145 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 468151435266 2026-02-13T02:14:57.146 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:14:57.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 468151435266 -lt 468151435266 2026-02-13T02:14:57.317 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:14:57.318 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:14:57.318 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:14:57.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:14:57.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:14:57.519 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:14:57.519 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:14:57.519 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:14:57.519 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:14:57.519 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:14:57.519 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:14:57.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:14:57.683 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:14:57.683 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:14:57.683 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:14:57.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:14:57.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:14:57.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:14:57.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1146: TEST_corrupt_scrub_replicated: echo -n val3-ROBJ8 2026-02-13T02:14:57.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1147: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2026-02-13T02:14:57.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:14:57.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:14:57.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:14:57.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:14:57.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2026-02-13T02:14:57.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:14:57.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:14:57.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:14:57.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:14:57.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:14:57.882 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:14:57.882 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:14:57.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:14:57.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:14:57.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:14:58.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:14:58.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2026-02-13T02:14:58.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:14:58.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:14:58.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:14:58.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:14:58.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:14:58.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2026-02-13T02:14:58.474 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-13T02:14:58.474 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:14:58.474 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:14:58.474 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:14:58.474 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:14:58.474 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:14:58.474 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:14:58.474 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:14:58.474 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:14:58.474 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:14:58.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:14:58.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:14:58.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:14:58.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:14:58.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:14:58.475 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:14:58.475 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:14:58.475 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:14:58.475 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:14:58.475 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:14:58.475 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:14:58.476 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:14:58.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:14:58.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:14:58.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:14:58.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:14:58.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:14:58.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:14:58.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:14:58.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:14:58.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:14:58.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:14:58.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:14:58.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:14:58.477 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:14:58.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:14:58.478 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:14:58.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:14:58.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:14:58.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:14:58.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:14:58.499 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:58.486+0000 7f7c21f4ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:14:58.501 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:58.486+0000 7f7c21f4ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:14:58.502 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:58.486+0000 7f7c21f4ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:14:58.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:14:58.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:14:58.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:14:58.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:14:58.642 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:14:58.642 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:14:58.642 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:14:58.642 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:14:58.642 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:14:58.642 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:14:58.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:14:58.908 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:58.894+0000 7f7c21f4ea40 -1 Falling back to public interface 2026-02-13T02:14:59.463 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:14:59.450+0000 7f7c21f4ea40 -1 osd.1 111 log_to_monitors true 2026-02-13T02:14:59.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:14:59.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:14:59.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:14:59.807 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:14:59.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:14:59.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:14:59.974 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 115 up_thru 115 down_at 112 last_clean_interval [109,111) [v2:127.0.0.1:6810/1582916383,v1:127.0.0.1:6811/1582916383] [v2:127.0.0.1:6812/1582916383,v1:127.0.0.1:6813/1582916383] exists,up cfe9e8e4-e309-485e-8178-7cb3d645195b 2026-02-13T02:14:59.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:14:59.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:14:59.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:14:59.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:14:59.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:14:59.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:14:59.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:14:59.975 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:14:59.976 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:14:59.976 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:14:59.976 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:14:59.976 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:14:59.976 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:15:00.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:15:00.079 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:15:00.079 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:15:00.079 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:15:00.079 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:15:00.079 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:15:00.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:15:00.241 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:15:00.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:15:00.242 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:15:00.242 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:15:00.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=442381631494 2026-02-13T02:15:00.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 442381631494 2026-02-13T02:15:00.315 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-442381631494' 2026-02-13T02:15:00.315 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:15:00.315 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:15:00.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=493921239042 2026-02-13T02:15:00.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 493921239042 2026-02-13T02:15:00.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-442381631494 1-493921239042' 2026-02-13T02:15:00.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:15:00.387 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-442381631494 2026-02-13T02:15:00.387 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:15:00.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:15:00.388 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-442381631494 2026-02-13T02:15:00.389 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:15:00.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=442381631494 2026-02-13T02:15:00.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 442381631494' 2026-02-13T02:15:00.390 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 442381631494 2026-02-13T02:15:00.391 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:15:00.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 442381631493 -lt 442381631494 2026-02-13T02:15:00.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:15:01.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:15:01.554 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:15:01.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 442381631494 -lt 442381631494 2026-02-13T02:15:01.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:15:01.714 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-493921239042 2026-02-13T02:15:01.714 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:15:01.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:15:01.716 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-493921239042 2026-02-13T02:15:01.716 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:15:01.718 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=493921239042 2026-02-13T02:15:01.718 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 493921239042' 2026-02-13T02:15:01.718 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 493921239042 2026-02-13T02:15:01.718 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:15:01.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 493921239042 -lt 493921239042 2026-02-13T02:15:01.891 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:15:01.892 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:15:01.892 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:15:02.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:15:02.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:15:02.082 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:15:02.082 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:15:02.082 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:15:02.082 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:15:02.082 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:15:02.082 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:15:02.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:15:02.240 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:15:02.240 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:15:02.241 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:15:02.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:15:02.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:15:02.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:15:02.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1148: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2026-02-13T02:15:02.441 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-13T02:15:02.441 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ9 2026-02-13T02:15:02.442 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 9 % 2 2026-02-13T02:15:02.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=1 2026-02-13T02:15:02.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-13T02:15:02.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1152: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 get-attr _ 2026-02-13T02:15:02.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:15:02.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:15:02.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:15:02.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:15:02.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 get-attr _ 2026-02-13T02:15:02.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:15:02.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:15:02.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:15:02.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:15:02.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:15:02.443 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:15:02.444 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:15:02.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:15:02.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:15:02.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:15:02.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:15:02.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 get-attr _ 2026-02-13T02:15:02.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:15:02.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:15:02.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:15:02.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:15:02.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:15:02.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 get-attr _ 2026-02-13T02:15:02.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-13T02:15:02.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:15:02.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:15:02.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:15:02.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:15:02.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:15:02.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:15:02.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:15:02.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:15:02.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:15:02.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:15:02.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:15:02.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:15:02.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:15:02.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:15:02.933 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:15:02.933 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:15:02.933 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:15:02.933 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:15:02.933 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:15:02.933 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:15:02.935 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:15:02.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:15:02.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:15:02.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:15:02.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:15:02.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:15:02.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:15:02.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:15:02.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:15:02.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:15:02.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:15:02.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:15:02.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:15:02.936 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:15:02.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:15:02.937 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:15:02.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:15:02.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:15:02.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:15:02.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:15:02.953 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:02.937+0000 7f9fbd051a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:15:02.954 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:02.941+0000 7f9fbd051a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:15:02.955 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:02.941+0000 7f9fbd051a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:15:03.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:15:03.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:15:03.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:15:03.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:15:03.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:15:03.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:15:03.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:15:03.091 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:15:03.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:15:03.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:15:03.258 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:15:03.308 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:03.293+0000 7f9fbd051a40 -1 Falling back to public interface 2026-02-13T02:15:03.878 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:03.865+0000 7f9fbd051a40 -1 osd.1 116 log_to_monitors true 2026-02-13T02:15:04.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:15:04.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:15:04.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:15:04.260 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:15:04.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:15:04.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:15:04.421 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 120 up_thru 120 down_at 117 last_clean_interval [115,116) [v2:127.0.0.1:6810/1676983482,v1:127.0.0.1:6811/1676983482] [v2:127.0.0.1:6812/1676983482,v1:127.0.0.1:6813/1676983482] exists,up cfe9e8e4-e309-485e-8178-7cb3d645195b 2026-02-13T02:15:04.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:15:04.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:15:04.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:15:04.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:15:04.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:15:04.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:15:04.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:15:04.421 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:15:04.422 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:15:04.422 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:15:04.422 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:15:04.423 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:15:04.423 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:15:04.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:15:04.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:15:04.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:15:04.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:15:04.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:15:04.533 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:15:04.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:15:04.696 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:15:04.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:15:04.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:15:04.697 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:15:04.770 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=442381631496 2026-02-13T02:15:04.771 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 442381631496 2026-02-13T02:15:04.771 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-442381631496' 2026-02-13T02:15:04.771 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:15:04.771 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:15:04.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=515396075522 2026-02-13T02:15:04.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 515396075522 2026-02-13T02:15:04.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-442381631496 1-515396075522' 2026-02-13T02:15:04.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:15:04.844 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-442381631496 2026-02-13T02:15:04.845 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:15:04.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:15:04.847 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-442381631496 2026-02-13T02:15:04.847 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:15:04.848 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=442381631496 2026-02-13T02:15:04.848 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 442381631496' 2026-02-13T02:15:04.849 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 442381631496 2026-02-13T02:15:04.849 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:15:05.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 442381631495 -lt 442381631496 2026-02-13T02:15:05.023 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:15:06.023 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:15:06.024 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:15:06.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 442381631495 -lt 442381631496 2026-02-13T02:15:06.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:15:07.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:15:07.187 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:15:07.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 442381631496 -lt 442381631496 2026-02-13T02:15:07.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:15:07.353 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-515396075522 2026-02-13T02:15:07.353 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:15:07.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:15:07.356 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-515396075522 2026-02-13T02:15:07.356 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:15:07.357 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=515396075522 2026-02-13T02:15:07.357 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 515396075522' 2026-02-13T02:15:07.357 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 515396075522 2026-02-13T02:15:07.358 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:15:07.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 515396075522 -lt 515396075522 2026-02-13T02:15:07.530 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:15:07.530 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:15:07.530 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:15:07.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:15:07.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:15:07.731 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:15:07.731 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:15:07.731 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:15:07.731 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:15:07.731 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:15:07.731 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:15:07.892 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:15:07.893 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:15:07.893 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:15:07.893 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:15:08.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:15:08.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:15:08.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:15:08.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1153: TEST_corrupt_scrub_replicated: echo -n D 2026-02-13T02:15:08.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1154: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2026-02-13T02:15:08.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1155: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2026-02-13T02:15:08.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:15:08.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:15:08.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:15:08.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:15:08.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2026-02-13T02:15:08.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:15:08.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:15:08.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:15:08.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:15:08.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:15:08.113 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:15:08.113 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:15:08.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:15:08.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:15:08.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:15:08.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:15:08.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2026-02-13T02:15:08.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:15:08.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:15:08.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:15:08.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:15:08.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:15:08.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2026-02-13T02:15:08.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-13T02:15:08.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:15:08.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:15:08.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:15:08.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:15:08.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:15:08.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:15:08.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:15:08.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:15:08.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:15:08.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:15:08.718 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:15:08.718 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:15:08.718 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:15:08.718 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:15:08.718 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:15:08.718 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:15:08.718 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:15:08.718 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:15:08.718 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:15:08.718 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:15:08.718 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:15:08.718 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:15:08.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:15:08.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:15:08.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:15:08.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:15:08.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:15:08.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:15:08.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:15:08.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:15:08.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:15:08.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:15:08.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:15:08.719 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:15:08.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:15:08.721 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:15:08.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:15:08.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:15:08.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:15:08.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:15:08.734 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:08.721+0000 7f377e65ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:15:08.736 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:08.721+0000 7f377e65ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:15:08.737 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:08.721+0000 7f377e65ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:15:08.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:15:08.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:15:08.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:15:08.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:15:08.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:15:08.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:15:08.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:15:08.882 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:15:08.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:15:08.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:15:09.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:15:09.156 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:09.141+0000 7f377e65ea40 -1 Falling back to public interface 2026-02-13T02:15:09.784 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:09.769+0000 7f377e65ea40 -1 osd.1 122 log_to_monitors true 2026-02-13T02:15:10.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:15:10.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:15:10.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:15:10.046 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:15:10.047 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:15:10.047 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:15:10.204 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 126 up_thru 126 down_at 123 last_clean_interval [120,122) [v2:127.0.0.1:6810/662496465,v1:127.0.0.1:6811/662496465] [v2:127.0.0.1:6812/662496465,v1:127.0.0.1:6813/662496465] exists,up cfe9e8e4-e309-485e-8178-7cb3d645195b 2026-02-13T02:15:10.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:15:10.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:15:10.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:15:10.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:15:10.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:15:10.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:15:10.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:15:10.205 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:15:10.206 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:15:10.206 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:15:10.206 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:15:10.206 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:15:10.206 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:15:10.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:15:10.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:15:10.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:15:10.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:15:10.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:15:10.328 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:15:10.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:15:10.483 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:15:10.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:15:10.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:15:10.483 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:15:10.557 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=442381631498 2026-02-13T02:15:10.557 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 442381631498 2026-02-13T02:15:10.557 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-442381631498' 2026-02-13T02:15:10.557 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:15:10.558 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:15:10.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=541165879298 2026-02-13T02:15:10.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 541165879298 2026-02-13T02:15:10.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-442381631498 1-541165879298' 2026-02-13T02:15:10.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:15:10.632 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-442381631498 2026-02-13T02:15:10.632 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:15:10.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:15:10.634 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-442381631498 2026-02-13T02:15:10.634 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:15:10.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=442381631498 2026-02-13T02:15:10.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 442381631498' 2026-02-13T02:15:10.636 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 442381631498 2026-02-13T02:15:10.636 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:15:10.795 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 442381631497 -lt 442381631498 2026-02-13T02:15:10.795 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:15:11.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:15:11.797 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:15:11.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 442381631497 -lt 442381631498 2026-02-13T02:15:11.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:15:12.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:15:12.959 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:15:13.121 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 442381631498 -lt 442381631498 2026-02-13T02:15:13.122 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:15:13.122 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-541165879298 2026-02-13T02:15:13.122 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:15:13.123 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:15:13.124 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-541165879298 2026-02-13T02:15:13.124 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:15:13.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=541165879298 2026-02-13T02:15:13.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 541165879298' 2026-02-13T02:15:13.126 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 541165879298 2026-02-13T02:15:13.127 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:15:13.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 541165879298 -lt 541165879298 2026-02-13T02:15:13.287 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:15:13.287 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:15:13.287 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:15:13.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:15:13.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:15:13.487 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:15:13.487 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:15:13.487 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:15:13.487 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:15:13.487 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:15:13.487 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:15:13.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:15:13.649 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:15:13.650 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:15:13.650 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:15:13.845 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:15:13.845 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:15:13.845 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:15:13.845 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1156: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2026-02-13T02:15:13.845 INFO:tasks.workunit.client.0.trial112.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2026-02-13T02:15:13.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-13T02:15:13.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ10 2026-02-13T02:15:13.846 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 10 % 2 2026-02-13T02:15:13.848 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=0 2026-02-13T02:15:13.848 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-13T02:15:13.848 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-13T02:15:13.848 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ11 2026-02-13T02:15:13.848 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 11 % 2 2026-02-13T02:15:13.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=1 2026-02-13T02:15:13.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-13T02:15:13.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-13T02:15:13.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ12 2026-02-13T02:15:13.850 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 12 % 2 2026-02-13T02:15:13.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=0 2026-02-13T02:15:13.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-13T02:15:13.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-13T02:15:13.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ13 2026-02-13T02:15:13.852 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 13 % 2 2026-02-13T02:15:13.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=1 2026-02-13T02:15:13.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-13T02:15:13.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-13T02:15:13.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ14 2026-02-13T02:15:13.853 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 14 % 2 2026-02-13T02:15:13.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=0 2026-02-13T02:15:13.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-13T02:15:13.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1165: TEST_corrupt_scrub_replicated: echo -n bad-val 2026-02-13T02:15:13.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1166: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2026-02-13T02:15:13.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:15:13.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:15:13.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T02:15:13.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:15:13.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2026-02-13T02:15:13.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:15:13.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:15:13.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T02:15:13.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:15:13.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T02:15:13.855 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:15:13.855 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:15:13.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:15:13.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:15:13.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:15:14.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:15:14.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2026-02-13T02:15:14.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:15:14.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:15:14.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:15:14.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:15:14.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:15:14.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2026-02-13T02:15:14.456 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-13T02:15:14.456 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:15:14.456 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:15:14.456 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:15:14.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:15:14.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:15:14.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:15:14.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:15:14.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:15:14.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:15:14.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:15:14.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:15:14.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:15:14.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:15:14.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:15:14.457 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:15:14.458 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:15:14.458 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:15:14.458 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:15:14.458 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:15:14.458 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:15:14.458 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:15:14.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:15:14.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:15:14.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:15:14.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:15:14.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:15:14.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:15:14.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:15:14.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:15:14.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:15:14.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:15:14.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:15:14.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:15:14.461 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T02:15:14.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:15:14.461 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T02:15:14.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:15:14.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:15:14.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:15:14.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:15:14.472 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:14.457+0000 7f77002dba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:15:14.474 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:14.457+0000 7f77002dba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:15:14.474 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:14.461+0000 7f77002dba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:15:14.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:15:14.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:15:14.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:15:14.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:15:14.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:15:14.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:15:14.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:15:14.617 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:15:14.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:15:14.618 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:15:14.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:15:14.899 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:14.885+0000 7f77002dba40 -1 Falling back to public interface 2026-02-13T02:15:15.553 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:15.537+0000 7f77002dba40 -1 osd.0 128 log_to_monitors true 2026-02-13T02:15:15.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:15:15.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:15:15.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:15:15.783 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:15:15.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:15:15.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:15:15.948 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 132 up_thru 132 down_at 129 last_clean_interval [103,128) [v2:127.0.0.1:6800/1323794222,v1:127.0.0.1:6801/1323794222] [v2:127.0.0.1:6802/1323794222,v1:127.0.0.1:6803/1323794222] exists,up 926506f0-916f-45ab-94cb-3aa841154594 2026-02-13T02:15:15.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:15:15.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:15:15.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:15:15.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:15:15.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:15:15.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:15:15.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:15:15.949 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:15:15.950 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:15:15.951 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:15:15.951 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:15:15.951 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:15:15.951 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:15:16.065 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:15:16.065 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:15:16.065 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:15:16.065 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:15:16.065 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:15:16.065 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:15:16.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:15:16.227 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:15:16.228 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:15:16.228 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:15:16.228 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:15:16.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=566935683074 2026-02-13T02:15:16.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 566935683074 2026-02-13T02:15:16.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-566935683074' 2026-02-13T02:15:16.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:15:16.305 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:15:16.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=541165879300 2026-02-13T02:15:16.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 541165879300 2026-02-13T02:15:16.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-566935683074 1-541165879300' 2026-02-13T02:15:16.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:15:16.379 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-566935683074 2026-02-13T02:15:16.379 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:15:16.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:15:16.381 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-566935683074 2026-02-13T02:15:16.381 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:15:16.383 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=566935683074 2026-02-13T02:15:16.383 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 566935683074' 2026-02-13T02:15:16.383 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 566935683074 2026-02-13T02:15:16.383 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:15:16.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 442381631499 -lt 566935683074 2026-02-13T02:15:16.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:15:17.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:15:17.545 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:15:17.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 566935683074 -lt 566935683074 2026-02-13T02:15:17.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:15:17.714 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-541165879300 2026-02-13T02:15:17.714 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:15:17.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:15:17.716 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-541165879300 2026-02-13T02:15:17.716 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:15:17.718 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=541165879300 2026-02-13T02:15:17.718 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 541165879300' 2026-02-13T02:15:17.718 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 541165879300 2026-02-13T02:15:17.719 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:15:17.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 541165879300 -lt 541165879300 2026-02-13T02:15:17.882 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:15:17.883 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:15:17.883 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:15:18.072 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:15:18.072 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:15:18.072 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:15:18.072 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:15:18.072 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:15:18.072 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:15:18.073 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:15:18.073 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:15:18.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:15:18.231 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:15:18.232 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:15:18.232 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:15:18.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:15:18.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:15:18.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:15:18.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1167: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2026-02-13T02:15:18.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:15:18.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:15:18.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:15:18.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:15:18.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2026-02-13T02:15:18.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:15:18.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:15:18.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:15:18.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:15:18.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:15:18.426 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:15:18.426 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:15:18.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:15:18.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:15:18.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:15:18.735 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:15:18.735 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2026-02-13T02:15:18.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:15:18.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:15:18.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:15:18.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:15:18.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:15:18.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ14 rm-attr _ 2026-02-13T02:15:19.042 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-13T02:15:19.042 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:15:19.042 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:15:19.042 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:15:19.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:15:19.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:15:19.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:15:19.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:15:19.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:15:19.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:15:19.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:15:19.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:15:19.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:15:19.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:15:19.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:15:19.043 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:15:19.043 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:15:19.043 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:15:19.043 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:15:19.043 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:15:19.043 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:15:19.044 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:15:19.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:15:19.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:15:19.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:15:19.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:15:19.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:15:19.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:15:19.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:15:19.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:15:19.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:15:19.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:15:19.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:15:19.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:15:19.045 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:15:19.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:15:19.046 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:15:19.047 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:15:19.047 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:15:19.047 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:15:19.047 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:15:19.064 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:19.049+0000 7fe0e129ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:15:19.066 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:19.053+0000 7fe0e129ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:15:19.067 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:19.053+0000 7fe0e129ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:15:19.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:15:19.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:15:19.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:15:19.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:15:19.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:15:19.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:15:19.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:15:19.205 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:15:19.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:15:19.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:15:19.369 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:15:19.483 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:19.469+0000 7fe0e129ba40 -1 Falling back to public interface 2026-02-13T02:15:20.050 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:20.037+0000 7fe0e129ba40 -1 osd.1 133 log_to_monitors true 2026-02-13T02:15:20.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:15:20.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:15:20.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:15:20.371 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:15:20.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:15:20.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:15:20.534 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 137 up_thru 137 down_at 134 last_clean_interval [126,133) [v2:127.0.0.1:6810/3973950521,v1:127.0.0.1:6811/3973950521] [v2:127.0.0.1:6812/3973950521,v1:127.0.0.1:6813/3973950521] exists,up cfe9e8e4-e309-485e-8178-7cb3d645195b 2026-02-13T02:15:20.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:15:20.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:15:20.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:15:20.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:15:20.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:15:20.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:15:20.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:15:20.536 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:15:20.536 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:15:20.536 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:15:20.536 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:15:20.536 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:15:20.536 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:15:20.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:15:20.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:15:20.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:15:20.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:15:20.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:15:20.653 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:15:20.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:15:20.817 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:15:20.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:15:20.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:15:20.818 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:15:20.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=566935683076 2026-02-13T02:15:20.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 566935683076 2026-02-13T02:15:20.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-566935683076' 2026-02-13T02:15:20.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:15:20.891 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:15:20.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=588410519554 2026-02-13T02:15:20.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 588410519554 2026-02-13T02:15:20.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-566935683076 1-588410519554' 2026-02-13T02:15:20.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:15:20.968 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-566935683076 2026-02-13T02:15:20.969 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:15:20.970 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:15:20.971 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-566935683076 2026-02-13T02:15:20.971 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:15:20.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=566935683076 2026-02-13T02:15:20.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 566935683076' 2026-02-13T02:15:20.973 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 566935683076 2026-02-13T02:15:20.973 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:15:21.144 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 566935683074 -lt 566935683076 2026-02-13T02:15:21.144 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:15:22.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:15:22.146 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:15:22.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 566935683074 -lt 566935683076 2026-02-13T02:15:22.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:15:23.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:15:23.307 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:15:23.474 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 566935683076 -lt 566935683076 2026-02-13T02:15:23.474 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:15:23.475 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-588410519554 2026-02-13T02:15:23.475 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:15:23.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:15:23.477 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-588410519554 2026-02-13T02:15:23.478 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:15:23.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=588410519554 2026-02-13T02:15:23.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 588410519554' 2026-02-13T02:15:23.479 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 588410519554 2026-02-13T02:15:23.480 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:15:23.646 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 588410519554 -lt 588410519554 2026-02-13T02:15:23.646 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:15:23.647 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:15:23.647 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:15:23.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:15:23.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:15:23.840 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:15:23.840 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:15:23.840 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:15:23.840 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:15:23.841 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:15:23.841 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:15:23.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:15:23.997 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:15:23.997 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:15:23.997 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:15:24.196 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:15:24.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:15:24.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:15:24.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1168: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val 2026-02-13T02:15:24.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-13T02:15:24.198 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ15 2026-02-13T02:15:24.198 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 15 % 2 2026-02-13T02:15:24.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=1 2026-02-13T02:15:24.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-13T02:15:24.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1172: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2026-02-13T02:15:24.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:15:24.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:15:24.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:15:24.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:15:24.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2026-02-13T02:15:24.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:15:24.201 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:15:24.201 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:15:24.201 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:15:24.201 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:15:24.201 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:15:24.201 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:15:24.202 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:15:24.202 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:15:24.202 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:15:24.510 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:15:24.511 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2026-02-13T02:15:24.511 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:15:24.511 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:15:24.511 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:15:24.511 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:15:24.511 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:15:24.511 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ15 rm-attr _ 2026-02-13T02:15:24.681 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2026-02-13T02:15:24.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-13T02:15:24.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:15:24.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:15:24.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:15:24.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:15:24.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:15:24.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:15:24.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:15:24.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:15:24.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:15:24.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:15:24.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:15:24.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:15:24.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:15:24.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:15:24.793 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:15:24.793 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:15:24.793 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:15:24.793 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:15:24.793 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:15:24.793 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:15:24.795 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:15:24.795 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:15:24.795 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:15:24.795 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:15:24.795 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:15:24.795 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:15:24.795 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:15:24.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:15:24.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:15:24.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:15:24.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:15:24.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:15:24.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:15:24.796 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:15:24.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:15:24.796 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:15:24.797 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:15:24.797 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:15:24.797 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:15:24.797 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:15:24.810 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:24.797+0000 7ff20d2eca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:15:24.811 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:24.797+0000 7ff20d2eca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:15:24.812 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:24.797+0000 7ff20d2eca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:15:24.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:15:24.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:15:24.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:15:24.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:15:24.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:15:24.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:15:24.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:15:24.961 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:15:24.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:15:24.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:15:25.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:15:25.203 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:25.189+0000 7ff20d2eca40 -1 Falling back to public interface 2026-02-13T02:15:25.817 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:25.802+0000 7ff20d2eca40 -1 osd.1 139 log_to_monitors true 2026-02-13T02:15:26.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:15:26.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:15:26.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:15:26.120 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:15:26.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:15:26.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:15:26.275 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 143 up_thru 143 down_at 140 last_clean_interval [137,139) [v2:127.0.0.1:6810/2091838350,v1:127.0.0.1:6811/2091838350] [v2:127.0.0.1:6812/2091838350,v1:127.0.0.1:6813/2091838350] exists,up cfe9e8e4-e309-485e-8178-7cb3d645195b 2026-02-13T02:15:26.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:15:26.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:15:26.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:15:26.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:15:26.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:15:26.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:15:26.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:15:26.277 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:15:26.277 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:15:26.277 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:15:26.278 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:15:26.278 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:15:26.278 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:15:26.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:15:26.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:15:26.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:15:26.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:15:26.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:15:26.392 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:15:26.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:15:26.552 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:15:26.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:15:26.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:15:26.552 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:15:26.625 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=566935683078 2026-02-13T02:15:26.625 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 566935683078 2026-02-13T02:15:26.625 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-566935683078' 2026-02-13T02:15:26.626 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:15:26.626 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:15:26.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=614180323330 2026-02-13T02:15:26.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 614180323330 2026-02-13T02:15:26.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-566935683078 1-614180323330' 2026-02-13T02:15:26.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:15:26.711 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-566935683078 2026-02-13T02:15:26.711 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:15:26.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:15:26.713 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-566935683078 2026-02-13T02:15:26.713 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:15:26.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=566935683078 2026-02-13T02:15:26.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 566935683078' 2026-02-13T02:15:26.715 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 566935683078 2026-02-13T02:15:26.715 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:15:26.877 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 566935683077 -lt 566935683078 2026-02-13T02:15:26.878 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:15:27.878 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:15:27.879 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:15:28.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 566935683077 -lt 566935683078 2026-02-13T02:15:28.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:15:29.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:15:29.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:15:29.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 566935683078 -lt 566935683078 2026-02-13T02:15:29.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:15:29.209 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-614180323330 2026-02-13T02:15:29.210 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:15:29.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:15:29.212 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-614180323330 2026-02-13T02:15:29.212 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:15:29.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=614180323330 2026-02-13T02:15:29.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 614180323330' 2026-02-13T02:15:29.214 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 614180323330 2026-02-13T02:15:29.214 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:15:29.372 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 614180323330 -lt 614180323330 2026-02-13T02:15:29.373 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:15:29.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:15:29.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:15:29.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:15:29.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:15:29.564 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:15:29.565 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:15:29.565 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:15:29.565 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:15:29.565 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:15:29.565 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:15:29.720 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:15:29.720 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:15:29.721 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:15:29.721 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:15:29.913 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:15:29.913 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:15:29.913 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:15:29.913 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-13T02:15:29.913 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ16 2026-02-13T02:15:29.913 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 16 % 2 2026-02-13T02:15:29.915 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=0 2026-02-13T02:15:29.915 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-13T02:15:29.915 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1176: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2026-02-13T02:15:29.915 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:15:29.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:15:29.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T02:15:29.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:15:29.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2026-02-13T02:15:29.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:15:29.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:15:29.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T02:15:29.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:15:29.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T02:15:29.916 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:15:29.916 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:15:29.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:15:29.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:15:29.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:15:30.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:15:30.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2026-02-13T02:15:30.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:15:30.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:15:30.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:15:30.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:15:30.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:15:30.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ16 rm-attr snapset 2026-02-13T02:15:30.397 INFO:tasks.workunit.client.0.trial112.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2026-02-13T02:15:30.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-13T02:15:30.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:15:30.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:15:30.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:15:30.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:15:30.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:15:30.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:15:30.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:15:30.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:15:30.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:15:30.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:15:30.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:15:30.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:15:30.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:15:30.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:15:30.494 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:15:30.494 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:15:30.494 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:15:30.494 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:15:30.494 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:15:30.495 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:15:30.495 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:15:30.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:15:30.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:15:30.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:15:30.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:15:30.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:15:30.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:15:30.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:15:30.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:15:30.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:15:30.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:15:30.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:15:30.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:15:30.496 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T02:15:30.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:15:30.497 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T02:15:30.497 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:15:30.497 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:15:30.497 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:15:30.497 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:15:30.513 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:30.498+0000 7fc91f3fca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:15:30.515 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:30.502+0000 7fc91f3fca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:15:30.515 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:30.502+0000 7fc91f3fca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:15:30.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:15:30.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:15:30.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:15:30.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:15:30.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:15:30.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:15:30.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:15:30.658 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:15:30.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:15:30.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:15:30.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:15:30.883 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:30.870+0000 7fc91f3fca40 -1 Falling back to public interface 2026-02-13T02:15:31.466 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:31.454+0000 7fc91f3fca40 -1 osd.0 145 log_to_monitors true 2026-02-13T02:15:31.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:15:31.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:15:31.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:15:31.820 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:15:31.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:15:31.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:15:31.982 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 149 up_thru 149 down_at 146 last_clean_interval [132,145) [v2:127.0.0.1:6800/3123791643,v1:127.0.0.1:6801/3123791643] [v2:127.0.0.1:6802/3123791643,v1:127.0.0.1:6803/3123791643] exists,up 926506f0-916f-45ab-94cb-3aa841154594 2026-02-13T02:15:31.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:15:31.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:15:31.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:15:31.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:15:31.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:15:31.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:15:31.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:15:31.983 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:15:31.983 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:15:31.984 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:15:31.984 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:15:31.984 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:15:31.984 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:15:32.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:15:32.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:15:32.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:15:32.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:15:32.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:15:32.098 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:15:32.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:15:32.259 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:15:32.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:15:32.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:15:32.260 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:15:32.336 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=639950127106 2026-02-13T02:15:32.336 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 639950127106 2026-02-13T02:15:32.336 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-639950127106' 2026-02-13T02:15:32.336 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:15:32.336 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:15:32.410 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=614180323332 2026-02-13T02:15:32.410 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 614180323332 2026-02-13T02:15:32.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-639950127106 1-614180323332' 2026-02-13T02:15:32.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:15:32.411 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-639950127106 2026-02-13T02:15:32.411 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:15:32.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:15:32.413 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-639950127106 2026-02-13T02:15:32.414 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:15:32.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=639950127106 2026-02-13T02:15:32.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 639950127106' 2026-02-13T02:15:32.415 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 639950127106 2026-02-13T02:15:32.416 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:15:32.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 566935683078 -lt 639950127106 2026-02-13T02:15:32.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:15:33.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:15:33.589 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:15:33.748 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 639950127106 -lt 639950127106 2026-02-13T02:15:33.748 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:15:33.749 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-614180323332 2026-02-13T02:15:33.749 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:15:33.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:15:33.751 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-614180323332 2026-02-13T02:15:33.752 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:15:33.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=614180323332 2026-02-13T02:15:33.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 614180323332' 2026-02-13T02:15:33.753 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 614180323332 2026-02-13T02:15:33.754 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:15:33.927 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 614180323332 -lt 614180323332 2026-02-13T02:15:33.927 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:15:33.928 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:15:33.928 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:15:34.128 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:15:34.128 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:15:34.129 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:15:34.129 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:15:34.129 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:15:34.129 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:15:34.129 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:15:34.129 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:15:34.291 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:15:34.292 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:15:34.292 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:15:34.293 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:15:34.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:15:34.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:15:34.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:15:34.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1177: TEST_corrupt_scrub_replicated: echo -n bad-val 2026-02-13T02:15:34.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1178: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2026-02-13T02:15:34.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:15:34.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:15:34.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:15:34.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:15:34.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2026-02-13T02:15:34.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:15:34.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:15:34.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:15:34.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:15:34.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:15:34.496 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:15:34.496 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:15:34.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:15:34.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:15:34.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:15:34.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:15:34.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2026-02-13T02:15:34.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:15:34.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:15:34.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:15:34.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:15:34.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:15:34.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2026-02-13T02:15:34.983 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2026-02-13T02:15:34.983 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2026-02-13T02:15:35.089 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-13T02:15:35.089 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:15:35.089 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:15:35.089 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:15:35.089 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:15:35.089 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:15:35.089 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:15:35.089 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:15:35.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:15:35.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:15:35.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:15:35.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:15:35.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:15:35.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:15:35.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:15:35.090 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:15:35.090 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:15:35.090 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:15:35.090 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:15:35.091 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:15:35.091 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:15:35.091 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:15:35.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:15:35.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:15:35.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:15:35.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:15:35.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:15:35.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:15:35.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:15:35.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:15:35.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:15:35.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:15:35.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:15:35.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:15:35.093 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:15:35.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:15:35.093 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:15:35.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:15:35.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:15:35.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:15:35.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:15:35.109 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:35.094+0000 7fedaa215a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:15:35.110 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:35.098+0000 7fedaa215a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:15:35.111 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:35.098+0000 7fedaa215a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:15:35.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:15:35.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:15:35.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:15:35.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:15:35.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:15:35.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:15:35.253 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:15:35.253 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:15:35.253 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:15:35.253 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:15:35.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:15:35.511 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:35.498+0000 7fedaa215a40 -1 Falling back to public interface 2026-02-13T02:15:36.167 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:36.154+0000 7fedaa215a40 -1 osd.1 150 log_to_monitors true 2026-02-13T02:15:36.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:15:36.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:15:36.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:15:36.419 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:15:36.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:15:36.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:15:36.581 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 154 up_thru 154 down_at 151 last_clean_interval [143,150) [v2:127.0.0.1:6810/2926483245,v1:127.0.0.1:6811/2926483245] [v2:127.0.0.1:6812/2926483245,v1:127.0.0.1:6813/2926483245] exists,up cfe9e8e4-e309-485e-8178-7cb3d645195b 2026-02-13T02:15:36.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:15:36.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:15:36.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:15:36.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:15:36.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:15:36.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:15:36.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:15:36.582 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:15:36.583 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:15:36.584 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:15:36.584 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:15:36.584 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:15:36.584 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:15:36.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:15:36.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:15:36.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:15:36.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:15:36.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:15:36.699 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:15:36.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:15:36.860 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:15:36.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:15:36.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:15:36.861 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:15:36.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=639950127108 2026-02-13T02:15:36.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 639950127108 2026-02-13T02:15:36.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-639950127108' 2026-02-13T02:15:36.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:15:36.937 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:15:37.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=661424963586 2026-02-13T02:15:37.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 661424963586 2026-02-13T02:15:37.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-639950127108 1-661424963586' 2026-02-13T02:15:37.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:15:37.010 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-639950127108 2026-02-13T02:15:37.010 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:15:37.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:15:37.012 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-639950127108 2026-02-13T02:15:37.012 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:15:37.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=639950127108 2026-02-13T02:15:37.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 639950127108' 2026-02-13T02:15:37.014 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 639950127108 2026-02-13T02:15:37.014 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:15:37.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 639950127106 -lt 639950127108 2026-02-13T02:15:37.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:15:38.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:15:38.174 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:15:38.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 639950127106 -lt 639950127108 2026-02-13T02:15:38.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:15:39.343 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:15:39.344 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:15:39.507 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 639950127108 -lt 639950127108 2026-02-13T02:15:39.508 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:15:39.508 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-661424963586 2026-02-13T02:15:39.509 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:15:39.510 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:15:39.511 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-661424963586 2026-02-13T02:15:39.511 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:15:39.512 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=661424963586 2026-02-13T02:15:39.513 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 661424963586' 2026-02-13T02:15:39.513 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 661424963586 2026-02-13T02:15:39.513 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:15:39.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 661424963586 -lt 661424963586 2026-02-13T02:15:39.674 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:15:39.675 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:15:39.675 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:15:39.869 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:15:39.870 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:15:39.871 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:15:39.871 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:15:39.871 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:15:39.871 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:15:39.871 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:15:39.871 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:15:40.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:15:40.027 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:15:40.027 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:15:40.028 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:15:40.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:15:40.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:15:40.217 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:15:40.217 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-13T02:15:40.217 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ17 2026-02-13T02:15:40.217 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 17 % 2 2026-02-13T02:15:40.219 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=1 2026-02-13T02:15:40.219 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-13T02:15:40.219 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1183: TEST_corrupt_scrub_replicated: local payload=ROBJ17 2026-02-13T02:15:40.219 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: echo ROBJ17 2026-02-13T02:15:40.219 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1185: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-13T02:15:40.219 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:15:40.219 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:15:40.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T02:15:40.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:15:40.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-13T02:15:40.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:15:40.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:15:40.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T02:15:40.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:15:40.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T02:15:40.220 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:15:40.220 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:15:40.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:15:40.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:15:40.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:15:40.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:15:40.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-13T02:15:40.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:15:40.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:15:40.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:15:40.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:15:40.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:15:40.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-13T02:15:40.715 INFO:tasks.workunit.client.0.trial112.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2026-02-13T02:15:40.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-13T02:15:40.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:15:40.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:15:40.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:15:40.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:15:40.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:15:40.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:15:40.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:15:40.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:15:40.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:15:40.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:15:40.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:15:40.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:15:40.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:15:40.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:15:40.823 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:15:40.823 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:15:40.823 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:15:40.823 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:15:40.823 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:15:40.823 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:15:40.823 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:15:40.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:15:40.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:15:40.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:15:40.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:15:40.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:15:40.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:15:40.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:15:40.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:15:40.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:15:40.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:15:40.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:15:40.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:15:40.825 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T02:15:40.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:15:40.826 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T02:15:40.827 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:15:40.827 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:15:40.827 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:15:40.827 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:15:40.845 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:40.830+0000 7f2410dd7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:15:40.846 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:40.834+0000 7f2410dd7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:15:40.847 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:40.834+0000 7f2410dd7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:15:40.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:15:40.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:15:40.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:15:40.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:15:40.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:15:40.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:15:40.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:15:40.985 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:15:40.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:15:40.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:15:41.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:15:41.219 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:41.206+0000 7f2410dd7a40 -1 Falling back to public interface 2026-02-13T02:15:41.783 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:41.770+0000 7f2410dd7a40 -1 osd.0 155 log_to_monitors true 2026-02-13T02:15:42.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:15:42.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:15:42.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:15:42.149 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:15:42.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:15:42.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:15:42.312 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 159 up_thru 159 down_at 156 last_clean_interval [149,155) [v2:127.0.0.1:6800/657034164,v1:127.0.0.1:6801/657034164] [v2:127.0.0.1:6802/657034164,v1:127.0.0.1:6803/657034164] exists,up 926506f0-916f-45ab-94cb-3aa841154594 2026-02-13T02:15:42.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:15:42.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:15:42.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:15:42.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:15:42.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:15:42.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:15:42.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:15:42.312 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:15:42.313 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:15:42.313 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:15:42.314 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:15:42.314 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:15:42.314 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:15:42.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:15:42.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:15:42.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:15:42.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:15:42.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:15:42.427 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:15:42.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:15:42.587 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:15:42.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:15:42.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:15:42.588 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:15:42.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=682899800066 2026-02-13T02:15:42.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 682899800066 2026-02-13T02:15:42.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-682899800066' 2026-02-13T02:15:42.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:15:42.664 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:15:42.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=661424963588 2026-02-13T02:15:42.737 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 661424963588 2026-02-13T02:15:42.737 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-682899800066 1-661424963588' 2026-02-13T02:15:42.737 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:15:42.737 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-682899800066 2026-02-13T02:15:42.737 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:15:42.739 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:15:42.740 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-682899800066 2026-02-13T02:15:42.740 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:15:42.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=682899800066 2026-02-13T02:15:42.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 682899800066' 2026-02-13T02:15:42.741 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 682899800066 2026-02-13T02:15:42.742 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:15:42.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 682899800066 2026-02-13T02:15:42.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:15:43.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:15:43.913 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:15:44.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 682899800066 2026-02-13T02:15:44.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:15:45.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:15:45.080 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:15:45.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 682899800066 -lt 682899800066 2026-02-13T02:15:45.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:15:45.248 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-661424963588 2026-02-13T02:15:45.248 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:15:45.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:15:45.250 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-661424963588 2026-02-13T02:15:45.250 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:15:45.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=661424963588 2026-02-13T02:15:45.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 661424963588' 2026-02-13T02:15:45.252 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 661424963588 2026-02-13T02:15:45.252 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:15:45.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 661424963588 -lt 661424963588 2026-02-13T02:15:45.419 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:15:45.420 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:15:45.420 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:15:45.610 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:15:45.611 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:15:45.611 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:15:45.611 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:15:45.611 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:15:45.611 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:15:45.611 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:15:45.611 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:15:45.775 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:15:45.775 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:15:45.776 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:15:45.776 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:15:45.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:15:45.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:15:45.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:15:45.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1186: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-13T02:15:45.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:15:45.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:15:45.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:15:45.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:15:45.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-13T02:15:45.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:15:45.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:15:45.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:15:45.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:15:45.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:15:45.978 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:15:45.978 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:15:45.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:15:45.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:15:45.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:15:46.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:15:46.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-13T02:15:46.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:15:46.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:15:46.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:15:46.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:15:46.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:15:46.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-13T02:15:46.478 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2026-02-13T02:15:46.478 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2026-02-13T02:15:46.568 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-13T02:15:46.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:15:46.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:15:46.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:15:46.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:15:46.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:15:46.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:15:46.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:15:46.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:15:46.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:15:46.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:15:46.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:15:46.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:15:46.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:15:46.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:15:46.570 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:15:46.570 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:15:46.570 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:15:46.570 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:15:46.570 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:15:46.571 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:15:46.571 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:15:46.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:15:46.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:15:46.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:15:46.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:15:46.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:15:46.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:15:46.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:15:46.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:15:46.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:15:46.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:15:46.572 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:15:46.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:15:46.573 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:15:46.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:15:46.573 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:15:46.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:15:46.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:15:46.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:15:46.574 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:15:46.591 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:46.578+0000 7f17e689da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:15:46.593 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:46.578+0000 7f17e689da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:15:46.594 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:46.582+0000 7f17e689da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:15:46.735 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:15:46.735 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:15:46.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:15:46.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:15:46.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:15:46.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:15:46.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:15:46.736 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:15:46.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:15:46.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:15:46.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:15:47.003 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:46.990+0000 7f17e689da40 -1 Falling back to public interface 2026-02-13T02:15:47.568 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:47.554+0000 7f17e689da40 -1 osd.1 161 log_to_monitors true 2026-02-13T02:15:47.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:15:47.898 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:15:47.898 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:15:47.898 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:15:47.898 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:15:47.898 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:15:48.059 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 165 up_thru 165 down_at 162 last_clean_interval [154,161) [v2:127.0.0.1:6810/2696096642,v1:127.0.0.1:6811/2696096642] [v2:127.0.0.1:6812/2696096642,v1:127.0.0.1:6813/2696096642] exists,up cfe9e8e4-e309-485e-8178-7cb3d645195b 2026-02-13T02:15:48.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:15:48.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:15:48.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:15:48.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:15:48.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:15:48.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:15:48.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:15:48.061 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:15:48.061 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:15:48.061 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:15:48.062 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:15:48.062 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:15:48.062 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:15:48.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:15:48.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:15:48.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:15:48.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:15:48.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:15:48.169 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:15:48.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:15:48.334 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:15:48.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:15:48.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:15:48.335 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:15:48.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=682899800068 2026-02-13T02:15:48.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 682899800068 2026-02-13T02:15:48.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-682899800068' 2026-02-13T02:15:48.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:15:48.409 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:15:48.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=708669603842 2026-02-13T02:15:48.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 708669603842 2026-02-13T02:15:48.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-682899800068 1-708669603842' 2026-02-13T02:15:48.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:15:48.481 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-682899800068 2026-02-13T02:15:48.481 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:15:48.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:15:48.483 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-682899800068 2026-02-13T02:15:48.484 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:15:48.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=682899800068 2026-02-13T02:15:48.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 682899800068' 2026-02-13T02:15:48.485 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 682899800068 2026-02-13T02:15:48.486 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:15:48.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 682899800068 -lt 682899800068 2026-02-13T02:15:48.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:15:48.649 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-708669603842 2026-02-13T02:15:48.649 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:15:48.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:15:48.651 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-708669603842 2026-02-13T02:15:48.651 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:15:48.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=708669603842 2026-02-13T02:15:48.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 708669603842' 2026-02-13T02:15:48.653 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 708669603842 2026-02-13T02:15:48.653 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:15:48.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 708669603842 -lt 708669603842 2026-02-13T02:15:48.822 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:15:48.822 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:15:48.822 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:15:49.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:15:49.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:15:49.020 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:15:49.020 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:15:49.020 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:15:49.020 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:15:49.020 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:15:49.020 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:15:49.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:15:49.179 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:15:49.180 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:15:49.180 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:15:49.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:15:49.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:15:49.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:15:49.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-13T02:15:49.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ18 2026-02-13T02:15:49.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 18 % 2 2026-02-13T02:15:49.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=0 2026-02-13T02:15:49.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-13T02:15:49.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1191: TEST_corrupt_scrub_replicated: local payload=ROBJ18 2026-02-13T02:15:49.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1192: TEST_corrupt_scrub_replicated: echo ROBJ18 2026-02-13T02:15:49.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1193: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-13T02:15:49.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:15:49.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:15:49.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T02:15:49.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:15:49.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-13T02:15:49.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:15:49.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:15:49.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T02:15:49.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:15:49.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T02:15:49.378 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:15:49.378 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:15:49.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:15:49.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:15:49.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:15:49.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:15:49.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-13T02:15:49.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:15:49.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:15:49.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:15:49.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:15:49.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:15:49.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-13T02:15:49.860 INFO:tasks.workunit.client.0.trial112.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2026-02-13T02:15:49.971 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-13T02:15:49.971 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:15:49.971 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:15:49.971 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:15:49.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:15:49.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:15:49.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:15:49.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:15:49.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:15:49.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:15:49.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:15:49.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:15:49.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:15:49.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:15:49.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:15:49.972 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:15:49.973 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:15:49.973 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:15:49.973 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:15:49.973 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:15:49.973 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:15:49.975 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:15:49.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:15:49.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:15:49.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:15:49.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:15:49.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:15:49.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:15:49.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:15:49.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:15:49.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:15:49.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:15:49.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:15:49.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:15:49.976 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T02:15:49.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:15:49.976 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T02:15:49.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:15:49.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:15:49.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:15:49.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:15:49.992 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:49.978+0000 7f576f585a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:15:49.993 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:49.982+0000 7f576f585a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:15:49.995 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:49.982+0000 7f576f585a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:15:50.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:15:50.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:15:50.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:15:50.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:15:50.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:15:50.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:15:50.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:15:50.140 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:15:50.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:15:50.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:15:50.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:15:50.431 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:50.418+0000 7f576f585a40 -1 Falling back to public interface 2026-02-13T02:15:51.007 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:50.994+0000 7f576f585a40 -1 osd.0 166 log_to_monitors true 2026-02-13T02:15:51.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:15:51.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:15:51.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:15:51.303 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:15:51.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:15:51.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:15:51.467 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 170 up_thru 170 down_at 167 last_clean_interval [159,166) [v2:127.0.0.1:6800/2380006122,v1:127.0.0.1:6801/2380006122] [v2:127.0.0.1:6802/2380006122,v1:127.0.0.1:6803/2380006122] exists,up 926506f0-916f-45ab-94cb-3aa841154594 2026-02-13T02:15:51.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:15:51.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:15:51.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:15:51.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:15:51.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:15:51.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:15:51.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:15:51.469 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:15:51.469 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:15:51.469 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:15:51.469 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:15:51.470 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:15:51.470 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:15:51.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:15:51.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:15:51.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:15:51.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:15:51.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:15:51.590 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:15:51.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:15:51.752 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:15:51.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:15:51.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:15:51.752 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:15:51.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=730144440322 2026-02-13T02:15:51.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 730144440322 2026-02-13T02:15:51.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-730144440322' 2026-02-13T02:15:51.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:15:51.830 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:15:51.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=708669603843 2026-02-13T02:15:51.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 708669603843 2026-02-13T02:15:51.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-730144440322 1-708669603843' 2026-02-13T02:15:51.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:15:51.904 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-730144440322 2026-02-13T02:15:51.904 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:15:51.905 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:15:51.906 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-730144440322 2026-02-13T02:15:51.906 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:15:51.908 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=730144440322 2026-02-13T02:15:51.908 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 730144440322' 2026-02-13T02:15:51.908 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 730144440322 2026-02-13T02:15:51.908 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:15:52.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 682899800068 -lt 730144440322 2026-02-13T02:15:52.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:15:53.079 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:15:53.079 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:15:53.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 730144440322 -lt 730144440322 2026-02-13T02:15:53.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:15:53.245 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-708669603843 2026-02-13T02:15:53.245 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:15:53.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:15:53.247 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-708669603843 2026-02-13T02:15:53.247 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:15:53.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=708669603843 2026-02-13T02:15:53.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 708669603843' 2026-02-13T02:15:53.249 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 708669603843 2026-02-13T02:15:53.249 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:15:53.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 708669603843 -lt 708669603843 2026-02-13T02:15:53.425 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:15:53.426 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:15:53.426 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:15:53.621 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:15:53.621 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:15:53.622 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:15:53.623 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:15:53.623 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:15:53.623 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:15:53.623 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:15:53.623 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:15:53.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:15:53.783 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:15:53.783 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:15:53.784 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:15:53.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:15:53.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:15:53.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:15:53.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1194: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-13T02:15:53.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:15:53.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:15:53.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:15:53.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:15:53.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-13T02:15:53.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:15:53.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:15:53.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:15:53.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:15:53.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:15:53.984 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:15:53.984 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:15:53.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:15:53.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:15:53.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:15:54.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:15:54.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-13T02:15:54.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:15:54.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:15:54.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:15:54.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:15:54.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:15:54.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-13T02:15:54.482 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2026-02-13T02:15:54.482 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2026-02-13T02:15:54.585 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-13T02:15:54.585 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:15:54.585 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:15:54.585 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:15:54.585 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:15:54.585 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:15:54.585 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:15:54.585 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:15:54.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:15:54.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:15:54.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:15:54.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:15:54.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:15:54.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:15:54.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:15:54.586 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:15:54.586 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:15:54.586 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:15:54.586 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:15:54.587 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:15:54.587 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:15:54.587 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:15:54.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:15:54.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:15:54.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:15:54.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:15:54.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:15:54.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:15:54.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:15:54.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:15:54.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:15:54.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:15:54.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:15:54.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:15:54.589 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:15:54.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:15:54.589 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:15:54.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:15:54.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:15:54.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:15:54.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:15:54.608 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:54.594+0000 7ff005d33a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:15:54.610 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:54.598+0000 7ff005d33a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:15:54.611 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:54.598+0000 7ff005d33a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:15:54.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:15:54.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:15:54.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:15:54.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:15:54.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:15:54.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:15:54.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:15:54.749 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:15:54.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:15:54.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:15:54.910 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:15:54.979 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:54.966+0000 7ff005d33a40 -1 Falling back to public interface 2026-02-13T02:15:55.578 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:55.566+0000 7ff005d33a40 -1 osd.1 172 log_to_monitors true 2026-02-13T02:15:55.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:15:55.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:15:55.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:15:55.912 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:15:55.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:15:55.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:15:56.075 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 176 up_thru 176 down_at 173 last_clean_interval [165,172) [v2:127.0.0.1:6810/427186249,v1:127.0.0.1:6811/427186249] [v2:127.0.0.1:6812/427186249,v1:127.0.0.1:6813/427186249] exists,up cfe9e8e4-e309-485e-8178-7cb3d645195b 2026-02-13T02:15:56.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:15:56.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:15:56.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:15:56.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:15:56.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:15:56.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:15:56.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:15:56.076 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:15:56.077 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:15:56.077 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:15:56.077 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:15:56.077 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:15:56.077 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:15:56.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:15:56.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:15:56.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:15:56.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:15:56.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:15:56.193 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:15:56.358 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:15:56.359 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:15:56.359 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:15:56.359 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:15:56.359 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:15:56.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=730144440324 2026-02-13T02:15:56.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 730144440324 2026-02-13T02:15:56.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-730144440324' 2026-02-13T02:15:56.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:15:56.434 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:15:56.509 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=755914244098 2026-02-13T02:15:56.509 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 755914244098 2026-02-13T02:15:56.510 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-730144440324 1-755914244098' 2026-02-13T02:15:56.510 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:15:56.510 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-730144440324 2026-02-13T02:15:56.510 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:15:56.511 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:15:56.512 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-730144440324 2026-02-13T02:15:56.512 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:15:56.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=730144440324 2026-02-13T02:15:56.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 730144440324' 2026-02-13T02:15:56.514 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 730144440324 2026-02-13T02:15:56.515 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:15:56.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 730144440324 -lt 730144440324 2026-02-13T02:15:56.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:15:56.672 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-755914244098 2026-02-13T02:15:56.672 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:15:56.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:15:56.675 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-755914244098 2026-02-13T02:15:56.675 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:15:56.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=755914244098 2026-02-13T02:15:56.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 755914244098' 2026-02-13T02:15:56.676 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 755914244098 2026-02-13T02:15:56.677 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:15:56.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 755914244098 -lt 755914244098 2026-02-13T02:15:56.850 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:15:56.850 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:15:56.851 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:15:57.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:15:57.047 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:15:57.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:15:57.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:15:57.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:15:57.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:15:57.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:15:57.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:15:57.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:15:57.208 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:15:57.208 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:15:57.208 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:15:57.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:15:57.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:15:57.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:15:57.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1196: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 corrupt-info 2026-02-13T02:15:57.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:15:57.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:15:57.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:15:57.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:15:57.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 corrupt-info 2026-02-13T02:15:57.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:15:57.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:15:57.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:15:57.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:15:57.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:15:57.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:15:57.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:15:57.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:15:57.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:15:57.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:15:57.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:15:57.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 corrupt-info 2026-02-13T02:15:57.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:15:57.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:15:57.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:15:57.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:15:57.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:15:57.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 corrupt-info 2026-02-13T02:15:57.882 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2026-02-13T02:15:57.882 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2026-02-13T02:15:57.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-13T02:15:57.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:15:57.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:15:57.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:15:57.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:15:57.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:15:57.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:15:57.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:15:57.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:15:57.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:15:57.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:15:57.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:15:57.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:15:57.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:15:57.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:15:57.994 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:15:57.994 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:15:57.994 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:15:57.994 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:15:57.995 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:15:57.995 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:15:57.995 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:15:57.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:15:57.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:15:57.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:15:57.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:15:57.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:15:57.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:15:57.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:15:57.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:15:57.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:15:57.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:15:57.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:15:57.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:15:57.997 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:15:57.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:15:57.998 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:15:57.998 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:15:57.998 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:15:57.998 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:15:57.998 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:15:58.010 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:57.998+0000 7f5171e08a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:15:58.011 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:57.998+0000 7f5171e08a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:15:58.012 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:57.998+0000 7f5171e08a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:15:58.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:15:58.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:15:58.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:15:58.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:15:58.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:15:58.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:15:58.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:15:58.157 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:15:58.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:15:58.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:15:58.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:15:58.411 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:58.398+0000 7f5171e08a40 -1 Falling back to public interface 2026-02-13T02:15:59.063 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:15:59.050+0000 7f5171e08a40 -1 osd.1 177 log_to_monitors true 2026-02-13T02:15:59.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:15:59.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:15:59.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:15:59.324 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:15:59.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:15:59.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:15:59.487 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 181 up_thru 181 down_at 178 last_clean_interval [176,177) [v2:127.0.0.1:6810/3037792758,v1:127.0.0.1:6811/3037792758] [v2:127.0.0.1:6812/3037792758,v1:127.0.0.1:6813/3037792758] exists,up cfe9e8e4-e309-485e-8178-7cb3d645195b 2026-02-13T02:15:59.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:15:59.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:15:59.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:15:59.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:15:59.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:15:59.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:15:59.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:15:59.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:15:59.489 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:15:59.489 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:15:59.489 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:15:59.489 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:15:59.489 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:15:59.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:15:59.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:15:59.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:15:59.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:15:59.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:15:59.606 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:15:59.770 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:15:59.770 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:15:59.770 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:15:59.770 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:15:59.770 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:15:59.845 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=730144440325 2026-02-13T02:15:59.845 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 730144440325 2026-02-13T02:15:59.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-730144440325' 2026-02-13T02:15:59.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:15:59.846 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:15:59.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=777389080578 2026-02-13T02:15:59.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 777389080578 2026-02-13T02:15:59.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-730144440325 1-777389080578' 2026-02-13T02:15:59.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:15:59.921 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-730144440325 2026-02-13T02:15:59.921 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:15:59.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:15:59.923 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-730144440325 2026-02-13T02:15:59.923 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:15:59.925 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=730144440325 2026-02-13T02:15:59.925 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 730144440325' 2026-02-13T02:15:59.925 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 730144440325 2026-02-13T02:15:59.926 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:16:00.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 730144440324 -lt 730144440325 2026-02-13T02:16:00.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:16:01.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:16:01.095 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:16:01.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 730144440325 -lt 730144440325 2026-02-13T02:16:01.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:16:01.262 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-777389080578 2026-02-13T02:16:01.262 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:16:01.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:16:01.264 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-777389080578 2026-02-13T02:16:01.264 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:16:01.266 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=777389080578 2026-02-13T02:16:01.266 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 777389080578' 2026-02-13T02:16:01.266 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 777389080578 2026-02-13T02:16:01.266 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:16:01.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 777389080578 -lt 777389080578 2026-02-13T02:16:01.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:16:01.434 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:16:01.434 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:16:01.626 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:16:01.627 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:16:01.628 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:16:01.628 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:16:01.628 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:16:01.628 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:16:01.628 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:16:01.628 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:16:01.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:16:01.788 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:16:01.788 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:16:01.788 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:16:01.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:16:01.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:16:01.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:16:01.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-13T02:16:01.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ19 2026-02-13T02:16:01.977 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 19 % 2 2026-02-13T02:16:01.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=1 2026-02-13T02:16:01.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-13T02:16:01.978 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1205: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2026-02-13T02:16:01.978 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=csr_pool 2026-02-13T02:16:01.978 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=ROBJ0 2026-02-13T02:16:01.979 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map csr_pool ROBJ0 2026-02-13T02:16:01.979 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T02:16:02.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1205: TEST_corrupt_scrub_replicated: local pg=3.0 2026-02-13T02:16:02.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1207: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=1048576 2026-02-13T02:16:02.212 INFO:tasks.workunit.client.0.trial112.stdout:osd.0: {} 2026-02-13T02:16:02.212 INFO:tasks.workunit.client.0.trial112.stderr:osd.0: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_max_object_size = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2026-02-13T02:16:02.222 INFO:tasks.workunit.client.0.trial112.stdout:osd.1: {} 2026-02-13T02:16:02.222 INFO:tasks.workunit.client.0.trial112.stderr:osd.1: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_max_object_size = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2026-02-13T02:16:02.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1209: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2026-02-13T02:16:02.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: local pooltype=rep 2026-02-13T02:16:02.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: shift 2026-02-13T02:16:02.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: local which=data 2026-02-13T02:16:02.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: shift 2026-02-13T02:16:02.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: local poolname=csr_pool 2026-02-13T02:16:02.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: shift 2026-02-13T02:16:02.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: local objname=ROBJ11 2026-02-13T02:16:02.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: shift 2026-02-13T02:16:02.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: local dir=td/osd-scrub-repair 2026-02-13T02:16:02.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: shift 2026-02-13T02:16:02.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: local shard_id=0 2026-02-13T02:16:02.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2492: inject_eio: shift 2026-02-13T02:16:02.235 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: get_osds csr_pool ROBJ11 2026-02-13T02:16:02.235 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2026-02-13T02:16:02.236 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ11 2026-02-13T02:16:02.236 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ11 2026-02-13T02:16:02.236 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-13T02:16:02.397 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-13T02:16:02.397 INFO:tasks.workunit.client.0.trial112.stderr:0' 2026-02-13T02:16:02.397 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2026-02-13T02:16:02.397 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: initial_osds=('1' '0') 2026-02-13T02:16:02.397 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local -a initial_osds 2026-02-13T02:16:02.397 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: local osd_id=1 2026-02-13T02:16:02.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2496: inject_eio: '[' rep '!=' ec ']' 2026-02-13T02:16:02.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2497: inject_eio: shard_id= 2026-02-13T02:16:02.398 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: cat td/osd-scrub-repair/1/type 2026-02-13T02:16:02.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: type=bluestore 2026-02-13T02:16:02.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2026-02-13T02:16:02.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-13T02:16:02.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2026-02-13T02:16:02.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-13T02:16:02.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-13T02:16:02.400 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-13T02:16:02.401 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2026-02-13T02:16:02.401 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-13T02:16:02.401 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-13T02:16:02.401 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:16:02.401 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:16:02.401 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:16:02.401 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-osd.1.asok 2026-02-13T02:16:02.402 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.35207/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2026-02-13T02:16:02.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-13T02:16:02.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: local loop=0 2026-02-13T02:16:02.463 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2503: inject_eio: grep -q Invalid 2026-02-13T02:16:02.464 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: get_asok_path osd.1 2026-02-13T02:16:02.464 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-13T02:16:02.464 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-13T02:16:02.464 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:16:02.464 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:16:02.464 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:16:02.465 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-osd.1.asok 2026-02-13T02:16:02.465 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: CEPH_ARGS= 2026-02-13T02:16:02.465 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: ceph --admin-daemon /tmp/ceph-asok.35207/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2026-02-13T02:16:02.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1210: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2026-02-13T02:16:02.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: local pooltype=rep 2026-02-13T02:16:02.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: shift 2026-02-13T02:16:02.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: local which=mdata 2026-02-13T02:16:02.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: shift 2026-02-13T02:16:02.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: local poolname=csr_pool 2026-02-13T02:16:02.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: shift 2026-02-13T02:16:02.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: local objname=ROBJ12 2026-02-13T02:16:02.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: shift 2026-02-13T02:16:02.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: local dir=td/osd-scrub-repair 2026-02-13T02:16:02.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: shift 2026-02-13T02:16:02.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: local shard_id=1 2026-02-13T02:16:02.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2492: inject_eio: shift 2026-02-13T02:16:02.525 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: get_osds csr_pool ROBJ12 2026-02-13T02:16:02.525 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2026-02-13T02:16:02.525 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ12 2026-02-13T02:16:02.526 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ12 2026-02-13T02:16:02.526 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-13T02:16:02.684 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-13T02:16:02.684 INFO:tasks.workunit.client.0.trial112.stderr:0' 2026-02-13T02:16:02.684 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2026-02-13T02:16:02.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: initial_osds=('1' '0') 2026-02-13T02:16:02.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local -a initial_osds 2026-02-13T02:16:02.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: local osd_id=0 2026-02-13T02:16:02.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2496: inject_eio: '[' rep '!=' ec ']' 2026-02-13T02:16:02.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2497: inject_eio: shard_id= 2026-02-13T02:16:02.686 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: cat td/osd-scrub-repair/0/type 2026-02-13T02:16:02.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: type=bluestore 2026-02-13T02:16:02.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2026-02-13T02:16:02.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-13T02:16:02.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=0 2026-02-13T02:16:02.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-13T02:16:02.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-13T02:16:02.687 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-13T02:16:02.688 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.0 2026-02-13T02:16:02.688 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-13T02:16:02.688 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-13T02:16:02.688 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:16:02.688 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:16:02.688 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:16:02.688 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-osd.0.asok 2026-02-13T02:16:02.688 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.35207/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2026-02-13T02:16:02.748 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-13T02:16:02.748 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: local loop=0 2026-02-13T02:16:02.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2503: inject_eio: grep -q Invalid 2026-02-13T02:16:02.750 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: get_asok_path osd.0 2026-02-13T02:16:02.750 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-13T02:16:02.750 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-13T02:16:02.750 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:16:02.750 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:16:02.750 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:16:02.751 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-osd.0.asok 2026-02-13T02:16:02.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: CEPH_ARGS= 2026-02-13T02:16:02.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: ceph --admin-daemon /tmp/ceph-asok.35207/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2026-02-13T02:16:02.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1211: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2026-02-13T02:16:02.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: local pooltype=rep 2026-02-13T02:16:02.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: shift 2026-02-13T02:16:02.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: local which=mdata 2026-02-13T02:16:02.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: shift 2026-02-13T02:16:02.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: local poolname=csr_pool 2026-02-13T02:16:02.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: shift 2026-02-13T02:16:02.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: local objname=ROBJ13 2026-02-13T02:16:02.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: shift 2026-02-13T02:16:02.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: local dir=td/osd-scrub-repair 2026-02-13T02:16:02.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: shift 2026-02-13T02:16:02.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: local shard_id=1 2026-02-13T02:16:02.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2492: inject_eio: shift 2026-02-13T02:16:02.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: get_osds csr_pool ROBJ13 2026-02-13T02:16:02.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2026-02-13T02:16:02.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ13 2026-02-13T02:16:02.811 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ13 2026-02-13T02:16:02.812 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-13T02:16:02.974 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-13T02:16:02.974 INFO:tasks.workunit.client.0.trial112.stderr:0' 2026-02-13T02:16:02.974 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2026-02-13T02:16:02.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: initial_osds=('1' '0') 2026-02-13T02:16:02.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local -a initial_osds 2026-02-13T02:16:02.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: local osd_id=0 2026-02-13T02:16:02.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2496: inject_eio: '[' rep '!=' ec ']' 2026-02-13T02:16:02.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2497: inject_eio: shard_id= 2026-02-13T02:16:02.976 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: cat td/osd-scrub-repair/0/type 2026-02-13T02:16:02.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: type=bluestore 2026-02-13T02:16:02.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2026-02-13T02:16:02.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-13T02:16:02.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=0 2026-02-13T02:16:02.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-13T02:16:02.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-13T02:16:02.978 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-13T02:16:02.978 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.0 2026-02-13T02:16:02.978 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-13T02:16:02.978 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-13T02:16:02.978 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:16:02.978 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:16:02.979 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:16:02.979 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-osd.0.asok 2026-02-13T02:16:02.979 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.35207/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2026-02-13T02:16:03.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-13T02:16:03.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: local loop=0 2026-02-13T02:16:03.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2503: inject_eio: grep -q Invalid 2026-02-13T02:16:03.040 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: get_asok_path osd.0 2026-02-13T02:16:03.040 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-13T02:16:03.040 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-13T02:16:03.041 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:16:03.042 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:16:03.042 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:16:03.042 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-osd.0.asok 2026-02-13T02:16:03.042 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: CEPH_ARGS= 2026-02-13T02:16:03.042 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: ceph --admin-daemon /tmp/ceph-asok.35207/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2026-02-13T02:16:03.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1212: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2026-02-13T02:16:03.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: local pooltype=rep 2026-02-13T02:16:03.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: shift 2026-02-13T02:16:03.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: local which=data 2026-02-13T02:16:03.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: shift 2026-02-13T02:16:03.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: local poolname=csr_pool 2026-02-13T02:16:03.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: shift 2026-02-13T02:16:03.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: local objname=ROBJ13 2026-02-13T02:16:03.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: shift 2026-02-13T02:16:03.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: local dir=td/osd-scrub-repair 2026-02-13T02:16:03.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: shift 2026-02-13T02:16:03.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: local shard_id=0 2026-02-13T02:16:03.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2492: inject_eio: shift 2026-02-13T02:16:03.102 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: get_osds csr_pool ROBJ13 2026-02-13T02:16:03.102 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2026-02-13T02:16:03.102 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ13 2026-02-13T02:16:03.103 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ13 2026-02-13T02:16:03.103 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-13T02:16:03.270 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-13T02:16:03.271 INFO:tasks.workunit.client.0.trial112.stderr:0' 2026-02-13T02:16:03.271 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2026-02-13T02:16:03.271 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: initial_osds=('1' '0') 2026-02-13T02:16:03.271 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local -a initial_osds 2026-02-13T02:16:03.271 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: local osd_id=1 2026-02-13T02:16:03.271 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2496: inject_eio: '[' rep '!=' ec ']' 2026-02-13T02:16:03.271 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2497: inject_eio: shard_id= 2026-02-13T02:16:03.271 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: cat td/osd-scrub-repair/1/type 2026-02-13T02:16:03.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: type=bluestore 2026-02-13T02:16:03.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2026-02-13T02:16:03.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-13T02:16:03.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2026-02-13T02:16:03.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-13T02:16:03.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-13T02:16:03.273 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-13T02:16:03.274 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2026-02-13T02:16:03.274 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-13T02:16:03.274 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-13T02:16:03.274 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:16:03.274 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:16:03.275 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:16:03.275 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-osd.1.asok 2026-02-13T02:16:03.275 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.35207/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2026-02-13T02:16:03.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-13T02:16:03.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: local loop=0 2026-02-13T02:16:03.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2503: inject_eio: grep -q Invalid 2026-02-13T02:16:03.335 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: get_asok_path osd.1 2026-02-13T02:16:03.335 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-13T02:16:03.335 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-13T02:16:03.336 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:16:03.336 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:16:03.336 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:16:03.336 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-osd.1.asok 2026-02-13T02:16:03.336 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: CEPH_ARGS= 2026-02-13T02:16:03.336 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: ceph --admin-daemon /tmp/ceph-asok.35207/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2026-02-13T02:16:03.395 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1214: TEST_corrupt_scrub_replicated: pg_scrub 3.0 2026-02-13T02:16:03.395 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=3.0 2026-02-13T02:16:03.395 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 3.0 2026-02-13T02:16:03.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-13T02:16:03.397 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-13T02:16:03.397 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:16:03.397 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:16:03.397 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:16:03.397 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:16:03.397 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:16:03.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-13T02:16:03.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-13T02:16:03.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-13T02:16:03.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-13T02:16:03.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:16:03.702 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:16:03.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:16:03.865 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:16:03.866 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:16:03.866 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:16:03.866 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:16:03.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=730144440327 2026-02-13T02:16:03.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 730144440327 2026-02-13T02:16:03.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-730144440327' 2026-02-13T02:16:03.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:16:03.941 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:16:04.016 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=777389080579 2026-02-13T02:16:04.016 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 777389080579 2026-02-13T02:16:04.016 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-730144440327 1-777389080579' 2026-02-13T02:16:04.016 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:16:04.017 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-730144440327 2026-02-13T02:16:04.017 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:16:04.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:16:04.019 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-730144440327 2026-02-13T02:16:04.019 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:16:04.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=730144440327 2026-02-13T02:16:04.021 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 730144440327 2026-02-13T02:16:04.021 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 730144440327' 2026-02-13T02:16:04.021 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:16:04.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 730144440326 -lt 730144440327 2026-02-13T02:16:04.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:16:05.180 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:16:05.181 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:16:05.348 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 730144440327 -lt 730144440327 2026-02-13T02:16:05.348 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:16:05.349 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-777389080579 2026-02-13T02:16:05.349 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:16:05.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:16:05.351 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-777389080579 2026-02-13T02:16:05.352 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:16:05.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=777389080579 2026-02-13T02:16:05.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 777389080579' 2026-02-13T02:16:05.354 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 777389080579 2026-02-13T02:16:05.354 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:16:05.513 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 777389080580 -lt 777389080579 2026-02-13T02:16:05.513 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-13T02:16:05.513 INFO:tasks.workunit.client.0.trial112.stdout:#---------- 3.0 loop 0 2026-02-13T02:16:05.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-13T02:16:05.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-13T02:16:05.515 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-13T02:16:05.515 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-13T02:16:05.515 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-13T02:16:05.515 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-13T02:16:05.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-13T02:16:05.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-13T02:16:05.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-13T02:16:05.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-13T02:16:05.592 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: get_last_scrub_stamp 3.0 2026-02-13T02:16:05.592 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:16:05.592 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:16:05.592 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:16:05.592 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:16:05.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: local last_scrub=2026-02-13T02:14:07.964003+0000 2026-02-13T02:16:05.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_scrub: ceph pg scrub 3.0 2026-02-13T02:16:05.896 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 3.0 on osd.1 to scrub 2026-02-13T02:16:05.906 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_scrub: wait_for_scrub 3.0 2026-02-13T02:14:07.964003+0000 2026-02-13T02:16:05.906 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-13T02:16:05.906 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:14:07.964003+0000 2026-02-13T02:16:05.906 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:16:05.906 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:16:05.906 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:16:05.906 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-13T02:16:05.907 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:16:05.907 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:16:05.907 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:16:05.907 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:16:06.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:14:07.964003+0000 '>' 2026-02-13T02:14:07.964003+0000 2026-02-13T02:16:06.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:16:07.072 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:16:07.072 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:16:07.073 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-13T02:16:07.073 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:16:07.073 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:16:07.074 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:16:07.074 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:16:07.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:14:07.964003+0000 '>' 2026-02-13T02:14:07.964003+0000 2026-02-13T02:16:07.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:16:08.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:16:08.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:16:08.239 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-13T02:16:08.239 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:16:08.239 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:16:08.239 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:16:08.239 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:16:08.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:14:07.964003+0000 '>' 2026-02-13T02:14:07.964003+0000 2026-02-13T02:16:08.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:16:09.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:16:09.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:16:09.401 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-13T02:16:09.401 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:16:09.401 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:16:09.402 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:16:09.402 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:16:09.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:14:07.964003+0000 '>' 2026-02-13T02:14:07.964003+0000 2026-02-13T02:16:09.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:16:10.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:16:10.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:16:10.561 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-13T02:16:10.561 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:16:10.562 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:16:10.562 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:16:10.562 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:16:10.720 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:16:06.017065+0000 '>' 2026-02-13T02:14:07.964003+0000 2026-02-13T02:16:10.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:16:10.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1216: TEST_corrupt_scrub_replicated: ERRORS=0 2026-02-13T02:16:10.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1217: TEST_corrupt_scrub_replicated: declare -a s_err_strings 2026-02-13T02:16:10.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1218: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2026-02-13T02:16:10.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1219: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' 2026-02-13T02:16:10.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1220: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2026-02-13T02:16:10.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1221: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2026-02-13T02:16:10.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1222: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2026-02-13T02:16:10.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1223: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2026-02-13T02:16:10.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1224: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2026-02-13T02:16:10.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1225: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2026-02-13T02:16:10.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1226: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2026-02-13T02:16:10.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1227: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2026-02-13T02:16:10.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1228: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2026-02-13T02:16:10.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1229: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2026-02-13T02:16:10.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1230: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2026-02-13T02:16:10.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1231: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2026-02-13T02:16:10.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1232: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' 2026-02-13T02:16:10.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1233: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2026-02-13T02:16:10.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1234: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2026-02-13T02:16:10.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1235: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* v=3 cannot decode .* Malformed input' 2026-02-13T02:16:10.723 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1236: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2026-02-13T02:16:10.723 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1237: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' 2026-02-13T02:16:10.723 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1238: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' 2026-02-13T02:16:10.723 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1239: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' 2026-02-13T02:16:10.723 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:10.723 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:10.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:10.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:10.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:10.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:10.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:10.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:10.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:10.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:10.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:10.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:10.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:10.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:10.769 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:10.769 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:10.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:10.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:10.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:10.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:10.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:10.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:10.797 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:10.797 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:10.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:10.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:10.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:10.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:10.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:10.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:10.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:10.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:10.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:10.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:10.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:10.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* v=3 cannot decode .* Malformed input' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:10.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:10.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:10.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:10.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:10.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:10.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:10.871 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:10.872 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:10.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1250: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2026-02-13T02:16:10.905 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1252: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2026-02-13T02:16:10.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1252: TEST_corrupt_scrub_replicated: test 1 = 1 2026-02-13T02:16:10.921 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1254: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2026-02-13T02:16:10.929 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1254: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2026-02-13T02:16:10.929 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1256: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2026-02-13T02:16:10.948 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1258: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2026-02-13T02:16:10.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1258: TEST_corrupt_scrub_replicated: epoch=181 2026-02-13T02:16:10.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1260: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2026-02-13T02:16:10.958 INFO:tasks.workunit.client.0.trial112.stderr: . as $in 2026-02-13T02:16:10.958 INFO:tasks.workunit.client.0.trial112.stderr: | if type == "object" then 2026-02-13T02:16:10.958 INFO:tasks.workunit.client.0.trial112.stderr: reduce keys[] as $key 2026-02-13T02:16:10.958 INFO:tasks.workunit.client.0.trial112.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-13T02:16:10.958 INFO:tasks.workunit.client.0.trial112.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-13T02:16:10.958 INFO:tasks.workunit.client.0.trial112.stderr: else f 2026-02-13T02:16:10.958 INFO:tasks.workunit.client.0.trial112.stderr: end; 2026-02-13T02:16:10.958 INFO:tasks.workunit.client.0.trial112.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-13T02:16:10.959 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-13T02:16:10.959 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-13T02:16:10.959 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-13T02:16:10.959 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-13T02:16:10.959 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1260: TEST_corrupt_scrub_replicated: jq .inconsistents 2026-02-13T02:16:10.959 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1260: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-13T02:16:10.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2026-02-13T02:16:10.978 INFO:tasks.workunit.client.0.trial112.stderr: . as $in 2026-02-13T02:16:10.978 INFO:tasks.workunit.client.0.trial112.stderr: | if type == "object" then 2026-02-13T02:16:10.978 INFO:tasks.workunit.client.0.trial112.stderr: reduce keys[] as $key 2026-02-13T02:16:10.978 INFO:tasks.workunit.client.0.trial112.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-13T02:16:10.979 INFO:tasks.workunit.client.0.trial112.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-13T02:16:10.979 INFO:tasks.workunit.client.0.trial112.stderr: else f 2026-02-13T02:16:10.979 INFO:tasks.workunit.client.0.trial112.stderr: end; 2026-02-13T02:16:10.979 INFO:tasks.workunit.client.0.trial112.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-13T02:16:10.979 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-13T02:16:10.979 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-13T02:16:10.979 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-13T02:16:10.979 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2026-02-13T02:16:10.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: jq .inconsistents 2026-02-13T02:16:10.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-13T02:16:10.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2174: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-13T02:16:10.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2513: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-13T02:16:10.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: test no = yes 2026-02-13T02:16:10.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2180: TEST_corrupt_scrub_replicated: test '' = yes 2026-02-13T02:16:10.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2185: TEST_corrupt_scrub_replicated: objname=ROBJ9 2026-02-13T02:16:10.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2187: TEST_corrupt_scrub_replicated: echo -n ZZZ 2026-02-13T02:16:10.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2188: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2026-02-13T02:16:11.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2190: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2026-02-13T02:16:11.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:16:11.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:16:11.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T02:16:11.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:16:11.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2026-02-13T02:16:11.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:16:11.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:16:11.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T02:16:11.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:16:11.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T02:16:11.019 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:16:11.019 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:16:11.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:16:11.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:16:11.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:16:11.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:16:11.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2026-02-13T02:16:11.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:16:11.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:16:11.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:16:11.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:16:11.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:16:11.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2026-02-13T02:16:11.521 INFO:tasks.workunit.client.0.trial112.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2026-02-13T02:16:11.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-13T02:16:11.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:16:11.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:16:11.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:16:11.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:16:11.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:16:11.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:16:11.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:16:11.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:16:11.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:16:11.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:16:11.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:16:11.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:16:11.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:16:11.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:16:11.615 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:16:11.615 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:16:11.615 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:16:11.615 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:16:11.615 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:16:11.615 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:16:11.615 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:16:11.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:16:11.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:16:11.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:16:11.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:16:11.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:16:11.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:16:11.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:16:11.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:16:11.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:16:11.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:16:11.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:16:11.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:16:11.617 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T02:16:11.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:16:11.617 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T02:16:11.618 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:16:11.619 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:16:11.619 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:16:11.619 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:16:11.636 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:16:11.622+0000 7f34e0261a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:16:11.638 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:16:11.626+0000 7f34e0261a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:16:11.639 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:16:11.626+0000 7f34e0261a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:16:11.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:16:11.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:16:11.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:16:11.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:16:11.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:16:11.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:16:11.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:16:11.784 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:16:11.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:16:11.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:16:11.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:16:12.083 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:16:12.070+0000 7f34e0261a40 -1 Falling back to public interface 2026-02-13T02:16:12.612 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:16:12.598+0000 7f34e0261a40 -1 osd.0 184 log_to_monitors true 2026-02-13T02:16:12.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:16:12.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:16:12.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:16:12.947 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:16:12.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:16:12.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:16:13.110 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 187 up_thru 187 down_at 184 last_clean_interval [170,184) [v2:127.0.0.1:6800/2852131939,v1:127.0.0.1:6801/2852131939] [v2:127.0.0.1:6802/2852131939,v1:127.0.0.1:6803/2852131939] exists,up 926506f0-916f-45ab-94cb-3aa841154594 2026-02-13T02:16:13.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:16:13.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:16:13.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:16:13.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:16:13.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:16:13.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:16:13.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:16:13.111 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:16:13.112 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:16:13.113 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:16:13.113 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:16:13.113 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:16:13.113 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:16:13.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:16:13.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:16:13.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:16:13.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:16:13.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:16:13.226 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:16:13.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:16:13.392 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:16:13.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:16:13.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:16:13.393 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:16:13.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=803158884354 2026-02-13T02:16:13.465 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 803158884354 2026-02-13T02:16:13.465 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-803158884354' 2026-02-13T02:16:13.465 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:16:13.465 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:16:13.537 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=777389080582 2026-02-13T02:16:13.537 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 777389080582 2026-02-13T02:16:13.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-803158884354 1-777389080582' 2026-02-13T02:16:13.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:16:13.538 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-803158884354 2026-02-13T02:16:13.538 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:16:13.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:16:13.540 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-803158884354 2026-02-13T02:16:13.541 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:16:13.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=803158884354 2026-02-13T02:16:13.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 803158884354' 2026-02-13T02:16:13.542 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 803158884354 2026-02-13T02:16:13.543 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:16:13.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 730144440328 -lt 803158884354 2026-02-13T02:16:13.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:16:14.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:16:14.716 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:16:14.876 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 803158884354 -lt 803158884354 2026-02-13T02:16:14.876 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:16:14.877 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-777389080582 2026-02-13T02:16:14.877 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:16:14.878 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:16:14.879 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-777389080582 2026-02-13T02:16:14.879 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:16:14.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=777389080582 2026-02-13T02:16:14.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 777389080582' 2026-02-13T02:16:14.881 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 777389080582 2026-02-13T02:16:14.882 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:16:15.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 777389080583 -lt 777389080582 2026-02-13T02:16:15.044 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:16:15.045 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:16:15.045 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:16:15.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:16:15.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:16:15.247 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:16:15.247 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:16:15.247 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:16:15.247 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:16:15.247 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:16:15.247 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:16:15.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:16:15.410 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:16:15.410 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:16:15.410 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:16:15.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:16:15.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:16:15.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:16:15.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2191: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2026-02-13T02:16:15.608 INFO:tasks.workunit.client.0.trial112.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2026-02-13T02:16:15.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2193: TEST_corrupt_scrub_replicated: objname=ROBJ10 2026-02-13T02:16:15.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2194: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 get-attr _ 2026-02-13T02:16:15.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:16:15.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:16:15.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:16:15.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:16:15.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 get-attr _ 2026-02-13T02:16:15.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:16:15.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:16:15.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:16:15.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:16:15.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:16:15.609 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:16:15.609 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:16:15.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:16:15.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:16:15.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:16:15.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:16:15.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 get-attr _ 2026-02-13T02:16:15.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:16:15.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:16:15.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:16:15.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:16:15.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:16:15.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 get-attr _ 2026-02-13T02:16:15.989 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2026-02-13T02:16:15.989 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2026-02-13T02:16:16.072 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-13T02:16:16.072 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:16:16.072 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:16:16.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:16:16.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:16:16.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:16:16.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:16:16.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:16:16.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:16:16.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:16:16.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:16:16.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:16:16.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:16:16.074 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:16:16.074 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:16:16.074 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:16:16.074 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:16:16.074 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:16:16.074 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:16:16.074 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:16:16.074 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:16:16.074 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:16:16.074 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:16:16.074 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:16:16.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:16:16.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:16:16.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:16:16.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:16:16.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:16:16.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:16:16.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:16:16.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:16:16.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:16:16.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:16:16.077 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:16:16.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:16:16.077 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:16:16.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:16:16.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:16:16.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:16:16.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:16:16.090 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:16:16.078+0000 7f6bf9767a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:16:16.092 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:16:16.078+0000 7f6bf9767a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:16:16.092 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:16:16.078+0000 7f6bf9767a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:16:16.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:16:16.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:16:16.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:16:16.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:16:16.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:16:16.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:16:16.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:16:16.241 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:16:16.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:16:16.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:16:16.403 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:16:16.478 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:16:16.466+0000 7f6bf9767a40 -1 Falling back to public interface 2026-02-13T02:16:17.095 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:16:17.082+0000 7f6bf9767a40 -1 osd.1 189 log_to_monitors true 2026-02-13T02:16:17.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:16:17.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:16:17.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:16:17.404 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:16:17.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:16:17.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:16:17.564 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 192 up_thru 192 down_at 189 last_clean_interval [181,189) [v2:127.0.0.1:6810/882555546,v1:127.0.0.1:6811/882555546] [v2:127.0.0.1:6812/882555546,v1:127.0.0.1:6813/882555546] exists,up cfe9e8e4-e309-485e-8178-7cb3d645195b 2026-02-13T02:16:17.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:16:17.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:16:17.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:16:17.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:16:17.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:16:17.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:16:17.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:16:17.565 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:16:17.566 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:16:17.566 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:16:17.566 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:16:17.567 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:16:17.567 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:16:17.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:16:17.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:16:17.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:16:17.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:16:17.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:16:17.683 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:16:17.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:16:17.851 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:16:17.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:16:17.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:16:17.852 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:16:17.925 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=803158884356 2026-02-13T02:16:17.925 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 803158884356 2026-02-13T02:16:17.925 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-803158884356' 2026-02-13T02:16:17.925 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:16:17.926 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:16:17.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=824633720834 2026-02-13T02:16:17.998 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 824633720834 2026-02-13T02:16:17.998 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-803158884356 1-824633720834' 2026-02-13T02:16:17.998 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:16:17.998 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-803158884356 2026-02-13T02:16:17.998 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:16:18.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:16:18.000 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-803158884356 2026-02-13T02:16:18.001 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:16:18.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=803158884356 2026-02-13T02:16:18.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 803158884356' 2026-02-13T02:16:18.003 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 803158884356 2026-02-13T02:16:18.003 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:16:18.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 803158884354 -lt 803158884356 2026-02-13T02:16:18.167 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:16:19.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:16:19.168 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:16:19.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 803158884356 -lt 803158884356 2026-02-13T02:16:19.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:16:19.327 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-824633720834 2026-02-13T02:16:19.328 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:16:19.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:16:19.330 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-824633720834 2026-02-13T02:16:19.330 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:16:19.332 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=824633720834 2026-02-13T02:16:19.332 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 824633720834' 2026-02-13T02:16:19.332 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 824633720834 2026-02-13T02:16:19.332 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:16:19.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 824633720834 -lt 824633720834 2026-02-13T02:16:19.504 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:16:19.504 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:16:19.504 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:16:19.704 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:16:19.704 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:16:19.704 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:16:19.704 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:16:19.704 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:16:19.704 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:16:19.704 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:16:19.705 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:16:19.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:16:19.868 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:16:19.868 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:16:19.868 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:16:20.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:16:20.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:16:20.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:16:20.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2195: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key2-ROBJ10 val2-ROBJ10 2026-02-13T02:16:20.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2196: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2026-02-13T02:16:20.089 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:16:20.089 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:16:20.089 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T02:16:20.089 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:16:20.089 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2026-02-13T02:16:20.089 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:16:20.089 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:16:20.089 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T02:16:20.089 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:16:20.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T02:16:20.090 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:16:20.090 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:16:20.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:16:20.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:16:20.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:16:20.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:16:20.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2026-02-13T02:16:20.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:16:20.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:16:20.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:16:20.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:16:20.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:16:20.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2026-02-13T02:16:20.565 INFO:tasks.workunit.client.0.trial112.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2026-02-13T02:16:20.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-13T02:16:20.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:16:20.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:16:20.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:16:20.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:16:20.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:16:20.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:16:20.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:16:20.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:16:20.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:16:20.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:16:20.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:16:20.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:16:20.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:16:20.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:16:20.671 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:16:20.671 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:16:20.671 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:16:20.672 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:16:20.672 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:16:20.672 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:16:20.672 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:16:20.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:16:20.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:16:20.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:16:20.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:16:20.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:16:20.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:16:20.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:16:20.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:16:20.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:16:20.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:16:20.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:16:20.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:16:20.673 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T02:16:20.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:16:20.673 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T02:16:20.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:16:20.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:16:20.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:16:20.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:16:20.692 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:16:20.678+0000 7f9b5b998a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:16:20.693 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:16:20.682+0000 7f9b5b998a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:16:20.694 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:16:20.682+0000 7f9b5b998a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:16:20.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:16:20.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:16:20.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:16:20.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:16:20.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:16:20.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:16:20.839 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:16:20.839 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:16:20.839 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:16:20.839 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:16:21.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:16:21.079 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:16:21.066+0000 7f9b5b998a40 -1 Falling back to public interface 2026-02-13T02:16:21.665 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:16:21.654+0000 7f9b5b998a40 -1 osd.0 194 log_to_monitors true 2026-02-13T02:16:22.004 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:16:22.004 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:16:22.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:16:22.005 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:16:22.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:16:22.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:16:22.163 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 198 up_thru 198 down_at 195 last_clean_interval [187,194) [v2:127.0.0.1:6800/886121168,v1:127.0.0.1:6801/886121168] [v2:127.0.0.1:6802/886121168,v1:127.0.0.1:6803/886121168] exists,up 926506f0-916f-45ab-94cb-3aa841154594 2026-02-13T02:16:22.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:16:22.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:16:22.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:16:22.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:16:22.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:16:22.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:16:22.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:16:22.164 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:16:22.165 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:16:22.165 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:16:22.165 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:16:22.165 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:16:22.166 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:16:22.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:16:22.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:16:22.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:16:22.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:16:22.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:16:22.283 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:16:22.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:16:22.450 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:16:22.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:16:22.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:16:22.450 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:16:22.523 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=850403524610 2026-02-13T02:16:22.523 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 850403524610 2026-02-13T02:16:22.523 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-850403524610' 2026-02-13T02:16:22.523 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:16:22.523 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:16:22.598 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=824633720836 2026-02-13T02:16:22.598 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 824633720836 2026-02-13T02:16:22.598 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-850403524610 1-824633720836' 2026-02-13T02:16:22.598 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:16:22.600 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-850403524610 2026-02-13T02:16:22.600 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:16:22.601 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:16:22.602 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-850403524610 2026-02-13T02:16:22.602 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:16:22.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=850403524610 2026-02-13T02:16:22.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 850403524610' 2026-02-13T02:16:22.604 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 850403524610 2026-02-13T02:16:22.605 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:16:22.774 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 850403524610 -lt 850403524610 2026-02-13T02:16:22.774 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:16:22.775 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-824633720836 2026-02-13T02:16:22.775 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:16:22.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:16:22.777 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-824633720836 2026-02-13T02:16:22.777 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:16:22.779 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=824633720836 2026-02-13T02:16:22.779 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 824633720836' 2026-02-13T02:16:22.779 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 824633720836 2026-02-13T02:16:22.780 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:16:22.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 824633720835 -lt 824633720836 2026-02-13T02:16:22.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:16:23.950 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:16:23.950 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:16:24.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 824633720835 -lt 824633720836 2026-02-13T02:16:24.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:16:25.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:16:25.115 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:16:25.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 824633720836 -lt 824633720836 2026-02-13T02:16:25.274 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:16:25.275 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:16:25.275 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:16:25.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:16:25.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:16:25.473 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:16:25.473 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:16:25.473 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:16:25.473 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:16:25.474 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:16:25.474 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:16:25.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:16:25.634 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:16:25.634 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:16:25.634 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:16:25.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:16:25.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:16:25.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:16:25.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2197: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2026-02-13T02:16:25.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:16:25.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:16:25.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:16:25.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:16:25.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2026-02-13T02:16:25.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:16:25.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:16:25.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:16:25.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:16:25.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:16:25.832 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:16:25.832 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:16:25.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:16:25.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:16:25.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:16:26.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:16:26.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2026-02-13T02:16:26.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:16:26.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:16:26.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:16:26.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:16:26.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:16:26.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2026-02-13T02:16:26.322 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2026-02-13T02:16:26.322 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2026-02-13T02:16:26.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-13T02:16:26.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:16:26.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:16:26.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:16:26.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:16:26.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:16:26.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:16:26.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:16:26.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:16:26.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:16:26.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:16:26.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:16:26.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:16:26.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:16:26.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:16:26.425 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:16:26.425 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:16:26.426 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:16:26.426 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:16:26.426 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:16:26.426 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:16:26.426 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:16:26.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:16:26.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:16:26.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:16:26.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:16:26.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:16:26.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:16:26.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:16:26.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:16:26.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:16:26.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:16:26.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:16:26.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:16:26.428 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:16:26.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:16:26.428 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:16:26.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:16:26.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:16:26.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:16:26.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:16:26.443 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:16:26.430+0000 7faa4e604a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:16:26.445 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:16:26.434+0000 7faa4e604a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:16:26.446 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:16:26.434+0000 7faa4e604a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:16:26.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:16:26.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:16:26.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:16:26.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:16:26.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:16:26.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:16:26.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:16:26.592 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:16:26.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:16:26.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:16:26.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:16:26.851 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:16:26.838+0000 7faa4e604a40 -1 Falling back to public interface 2026-02-13T02:16:27.495 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:16:27.482+0000 7faa4e604a40 -1 osd.1 199 log_to_monitors true 2026-02-13T02:16:27.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:16:27.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:16:27.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:16:27.757 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:16:27.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:16:27.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:16:27.920 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 203 up_thru 203 down_at 200 last_clean_interval [192,199) [v2:127.0.0.1:6810/783782971,v1:127.0.0.1:6811/783782971] [v2:127.0.0.1:6812/783782971,v1:127.0.0.1:6813/783782971] exists,up cfe9e8e4-e309-485e-8178-7cb3d645195b 2026-02-13T02:16:27.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:16:27.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:16:27.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:16:27.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:16:27.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:16:27.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:16:27.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:16:27.921 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:16:27.922 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:16:27.922 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:16:27.922 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:16:27.922 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:16:27.922 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:16:28.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:16:28.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:16:28.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:16:28.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:16:28.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:16:28.044 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:16:28.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:16:28.204 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:16:28.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:16:28.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:16:28.205 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:16:28.277 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=850403524612 2026-02-13T02:16:28.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 850403524612 2026-02-13T02:16:28.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-850403524612' 2026-02-13T02:16:28.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:16:28.278 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:16:28.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=871878361090 2026-02-13T02:16:28.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 871878361090 2026-02-13T02:16:28.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-850403524612 1-871878361090' 2026-02-13T02:16:28.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:16:28.353 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-850403524612 2026-02-13T02:16:28.353 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:16:28.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:16:28.355 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-850403524612 2026-02-13T02:16:28.356 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:16:28.357 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=850403524612 2026-02-13T02:16:28.357 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 850403524612' 2026-02-13T02:16:28.358 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 850403524612 2026-02-13T02:16:28.358 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:16:28.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 850403524610 -lt 850403524612 2026-02-13T02:16:28.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:16:29.515 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:16:29.515 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:16:29.681 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 850403524612 -lt 850403524612 2026-02-13T02:16:29.681 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:16:29.682 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-871878361090 2026-02-13T02:16:29.682 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:16:29.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:16:29.684 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-871878361090 2026-02-13T02:16:29.685 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:16:29.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=871878361090 2026-02-13T02:16:29.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 871878361090' 2026-02-13T02:16:29.686 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 871878361090 2026-02-13T02:16:29.687 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:16:29.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 871878361090 -lt 871878361090 2026-02-13T02:16:29.852 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:16:29.852 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:16:29.853 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:16:30.050 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:16:30.051 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:16:30.051 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:16:30.051 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:16:30.051 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:16:30.051 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:16:30.051 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:16:30.052 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:16:30.210 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:16:30.211 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:16:30.211 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:16:30.211 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:16:30.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:16:30.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:16:30.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:16:30.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2198: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi 2026-02-13T02:16:30.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2200: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2026-02-13T02:16:30.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: local pooltype=rep 2026-02-13T02:16:30.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: shift 2026-02-13T02:16:30.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: local which=data 2026-02-13T02:16:30.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: shift 2026-02-13T02:16:30.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: local poolname=csr_pool 2026-02-13T02:16:30.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: shift 2026-02-13T02:16:30.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: local objname=ROBJ11 2026-02-13T02:16:30.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: shift 2026-02-13T02:16:30.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: local dir=td/osd-scrub-repair 2026-02-13T02:16:30.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: shift 2026-02-13T02:16:30.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: local shard_id=0 2026-02-13T02:16:30.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2492: inject_eio: shift 2026-02-13T02:16:30.420 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: get_osds csr_pool ROBJ11 2026-02-13T02:16:30.420 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2026-02-13T02:16:30.420 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ11 2026-02-13T02:16:30.420 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ11 2026-02-13T02:16:30.421 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-13T02:16:30.588 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-13T02:16:30.588 INFO:tasks.workunit.client.0.trial112.stderr:0' 2026-02-13T02:16:30.588 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2026-02-13T02:16:30.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: initial_osds=('1' '0') 2026-02-13T02:16:30.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local -a initial_osds 2026-02-13T02:16:30.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: local osd_id=1 2026-02-13T02:16:30.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2496: inject_eio: '[' rep '!=' ec ']' 2026-02-13T02:16:30.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2497: inject_eio: shard_id= 2026-02-13T02:16:30.590 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: cat td/osd-scrub-repair/1/type 2026-02-13T02:16:30.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: type=bluestore 2026-02-13T02:16:30.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2026-02-13T02:16:30.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-13T02:16:30.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2026-02-13T02:16:30.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-13T02:16:30.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-13T02:16:30.591 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-13T02:16:30.592 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2026-02-13T02:16:30.592 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-13T02:16:30.592 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-13T02:16:30.592 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:16:30.592 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:16:30.592 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:16:30.592 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-osd.1.asok 2026-02-13T02:16:30.593 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.35207/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2026-02-13T02:16:30.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-13T02:16:30.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: local loop=0 2026-02-13T02:16:30.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2503: inject_eio: grep -q Invalid 2026-02-13T02:16:30.653 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: get_asok_path osd.1 2026-02-13T02:16:30.653 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-13T02:16:30.653 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-13T02:16:30.654 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:16:30.654 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:16:30.654 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:16:30.655 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-osd.1.asok 2026-02-13T02:16:30.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: CEPH_ARGS= 2026-02-13T02:16:30.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: ceph --admin-daemon /tmp/ceph-asok.35207/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2026-02-13T02:16:30.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2201: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2026-02-13T02:16:30.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: local pooltype=rep 2026-02-13T02:16:30.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: shift 2026-02-13T02:16:30.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: local which=mdata 2026-02-13T02:16:30.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: shift 2026-02-13T02:16:30.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: local poolname=csr_pool 2026-02-13T02:16:30.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: shift 2026-02-13T02:16:30.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: local objname=ROBJ12 2026-02-13T02:16:30.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: shift 2026-02-13T02:16:30.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: local dir=td/osd-scrub-repair 2026-02-13T02:16:30.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: shift 2026-02-13T02:16:30.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: local shard_id=1 2026-02-13T02:16:30.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2492: inject_eio: shift 2026-02-13T02:16:30.717 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: get_osds csr_pool ROBJ12 2026-02-13T02:16:30.717 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2026-02-13T02:16:30.717 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ12 2026-02-13T02:16:30.717 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ12 2026-02-13T02:16:30.717 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-13T02:16:30.881 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-13T02:16:30.881 INFO:tasks.workunit.client.0.trial112.stderr:0' 2026-02-13T02:16:30.881 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2026-02-13T02:16:30.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: initial_osds=('1' '0') 2026-02-13T02:16:30.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local -a initial_osds 2026-02-13T02:16:30.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: local osd_id=0 2026-02-13T02:16:30.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2496: inject_eio: '[' rep '!=' ec ']' 2026-02-13T02:16:30.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2497: inject_eio: shard_id= 2026-02-13T02:16:30.883 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: cat td/osd-scrub-repair/0/type 2026-02-13T02:16:30.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: type=bluestore 2026-02-13T02:16:30.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2026-02-13T02:16:30.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-13T02:16:30.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=0 2026-02-13T02:16:30.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-13T02:16:30.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-13T02:16:30.885 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-13T02:16:30.885 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.0 2026-02-13T02:16:30.885 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-13T02:16:30.885 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-13T02:16:30.885 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:16:30.885 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:16:30.885 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:16:30.886 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-osd.0.asok 2026-02-13T02:16:30.886 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.35207/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2026-02-13T02:16:30.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-13T02:16:30.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: local loop=0 2026-02-13T02:16:30.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2503: inject_eio: grep -q Invalid 2026-02-13T02:16:30.946 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: get_asok_path osd.0 2026-02-13T02:16:30.947 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-13T02:16:30.947 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-13T02:16:30.948 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:16:30.948 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:16:30.948 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:16:30.948 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-osd.0.asok 2026-02-13T02:16:30.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: CEPH_ARGS= 2026-02-13T02:16:30.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: ceph --admin-daemon /tmp/ceph-asok.35207/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2026-02-13T02:16:31.006 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2202: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2026-02-13T02:16:31.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: local pooltype=rep 2026-02-13T02:16:31.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: shift 2026-02-13T02:16:31.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: local which=mdata 2026-02-13T02:16:31.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: shift 2026-02-13T02:16:31.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: local poolname=csr_pool 2026-02-13T02:16:31.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: shift 2026-02-13T02:16:31.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: local objname=ROBJ13 2026-02-13T02:16:31.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: shift 2026-02-13T02:16:31.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: local dir=td/osd-scrub-repair 2026-02-13T02:16:31.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: shift 2026-02-13T02:16:31.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: local shard_id=1 2026-02-13T02:16:31.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2492: inject_eio: shift 2026-02-13T02:16:31.008 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: get_osds csr_pool ROBJ13 2026-02-13T02:16:31.008 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2026-02-13T02:16:31.008 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ13 2026-02-13T02:16:31.008 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ13 2026-02-13T02:16:31.009 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-13T02:16:31.172 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-13T02:16:31.172 INFO:tasks.workunit.client.0.trial112.stderr:0' 2026-02-13T02:16:31.172 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2026-02-13T02:16:31.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: initial_osds=('1' '0') 2026-02-13T02:16:31.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local -a initial_osds 2026-02-13T02:16:31.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: local osd_id=0 2026-02-13T02:16:31.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2496: inject_eio: '[' rep '!=' ec ']' 2026-02-13T02:16:31.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2497: inject_eio: shard_id= 2026-02-13T02:16:31.174 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: cat td/osd-scrub-repair/0/type 2026-02-13T02:16:31.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: type=bluestore 2026-02-13T02:16:31.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2026-02-13T02:16:31.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-13T02:16:31.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=0 2026-02-13T02:16:31.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-13T02:16:31.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-13T02:16:31.175 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-13T02:16:31.175 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.0 2026-02-13T02:16:31.175 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-13T02:16:31.176 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-13T02:16:31.176 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:16:31.176 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:16:31.176 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:16:31.176 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-osd.0.asok 2026-02-13T02:16:31.176 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.35207/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2026-02-13T02:16:31.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-13T02:16:31.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: local loop=0 2026-02-13T02:16:31.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2503: inject_eio: grep -q Invalid 2026-02-13T02:16:31.237 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: get_asok_path osd.0 2026-02-13T02:16:31.238 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-13T02:16:31.238 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-13T02:16:31.238 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:16:31.238 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:16:31.238 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:16:31.238 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-osd.0.asok 2026-02-13T02:16:31.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: CEPH_ARGS= 2026-02-13T02:16:31.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: ceph --admin-daemon /tmp/ceph-asok.35207/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2026-02-13T02:16:31.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2203: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2026-02-13T02:16:31.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: local pooltype=rep 2026-02-13T02:16:31.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: shift 2026-02-13T02:16:31.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: local which=data 2026-02-13T02:16:31.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: shift 2026-02-13T02:16:31.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: local poolname=csr_pool 2026-02-13T02:16:31.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: shift 2026-02-13T02:16:31.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: local objname=ROBJ13 2026-02-13T02:16:31.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: shift 2026-02-13T02:16:31.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: local dir=td/osd-scrub-repair 2026-02-13T02:16:31.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: shift 2026-02-13T02:16:31.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: local shard_id=0 2026-02-13T02:16:31.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2492: inject_eio: shift 2026-02-13T02:16:31.298 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: get_osds csr_pool ROBJ13 2026-02-13T02:16:31.298 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2026-02-13T02:16:31.298 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ13 2026-02-13T02:16:31.299 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ13 2026-02-13T02:16:31.299 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-13T02:16:31.464 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-13T02:16:31.464 INFO:tasks.workunit.client.0.trial112.stderr:0' 2026-02-13T02:16:31.464 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2026-02-13T02:16:31.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: initial_osds=('1' '0') 2026-02-13T02:16:31.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local -a initial_osds 2026-02-13T02:16:31.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: local osd_id=1 2026-02-13T02:16:31.465 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2496: inject_eio: '[' rep '!=' ec ']' 2026-02-13T02:16:31.465 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2497: inject_eio: shard_id= 2026-02-13T02:16:31.465 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: cat td/osd-scrub-repair/1/type 2026-02-13T02:16:31.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: type=bluestore 2026-02-13T02:16:31.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2026-02-13T02:16:31.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-13T02:16:31.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2026-02-13T02:16:31.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-13T02:16:31.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-13T02:16:31.466 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-13T02:16:31.467 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2026-02-13T02:16:31.468 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-13T02:16:31.468 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-13T02:16:31.468 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:16:31.468 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:16:31.468 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:16:31.468 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-osd.1.asok 2026-02-13T02:16:31.468 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.35207/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2026-02-13T02:16:31.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-13T02:16:31.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: local loop=0 2026-02-13T02:16:31.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2503: inject_eio: grep -q Invalid 2026-02-13T02:16:31.530 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: get_asok_path osd.1 2026-02-13T02:16:31.531 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-13T02:16:31.531 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-13T02:16:31.531 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:16:31.531 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:16:31.531 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:16:31.531 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-osd.1.asok 2026-02-13T02:16:31.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: CEPH_ARGS= 2026-02-13T02:16:31.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: ceph --admin-daemon /tmp/ceph-asok.35207/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2026-02-13T02:16:31.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2206: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=134217728 2026-02-13T02:16:31.657 INFO:tasks.workunit.client.0.trial112.stdout:osd.0: {} 2026-02-13T02:16:31.657 INFO:tasks.workunit.client.0.trial112.stderr:osd.0: osd_max_object_size = '' 2026-02-13T02:16:31.666 INFO:tasks.workunit.client.0.trial112.stdout:osd.1: {} 2026-02-13T02:16:31.667 INFO:tasks.workunit.client.0.trial112.stderr:osd.1: osd_max_object_size = '' 2026-02-13T02:16:31.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2208: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2026-02-13T02:16:31.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1932: pg_deep_scrub: local pgid=3.0 2026-02-13T02:16:31.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: pg_deep_scrub: wait_for_pg_clean 3.0 2026-02-13T02:16:31.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-13T02:16:31.675 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-13T02:16:31.675 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:16:31.675 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:16:31.676 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:16:31.676 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:16:31.676 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:16:31.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-13T02:16:31.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-13T02:16:31.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-13T02:16:31.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-13T02:16:31.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:16:31.977 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:16:32.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:16:32.135 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:16:32.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:16:32.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:16:32.135 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:16:32.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=850403524614 2026-02-13T02:16:32.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 850403524614 2026-02-13T02:16:32.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-850403524614' 2026-02-13T02:16:32.210 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:16:32.210 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:16:32.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=871878361091 2026-02-13T02:16:32.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 871878361091 2026-02-13T02:16:32.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-850403524614 1-871878361091' 2026-02-13T02:16:32.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:16:32.282 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-850403524614 2026-02-13T02:16:32.282 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:16:32.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:16:32.285 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-850403524614 2026-02-13T02:16:32.285 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:16:32.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=850403524614 2026-02-13T02:16:32.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 850403524614' 2026-02-13T02:16:32.287 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 850403524614 2026-02-13T02:16:32.287 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:16:32.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 850403524612 -lt 850403524614 2026-02-13T02:16:32.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:16:33.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:16:33.462 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:16:33.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 850403524614 -lt 850403524614 2026-02-13T02:16:33.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:16:33.623 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-871878361091 2026-02-13T02:16:33.624 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:16:33.625 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:16:33.626 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-871878361091 2026-02-13T02:16:33.626 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:16:33.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=871878361091 2026-02-13T02:16:33.628 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 871878361091 2026-02-13T02:16:33.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 871878361091' 2026-02-13T02:16:33.628 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:16:33.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 871878361091 -lt 871878361091 2026-02-13T02:16:33.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-13T02:16:33.794 INFO:tasks.workunit.client.0.trial112.stdout:#---------- 3.0 loop 0 2026-02-13T02:16:33.795 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-13T02:16:33.795 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-13T02:16:33.795 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-13T02:16:33.795 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-13T02:16:33.795 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-13T02:16:33.795 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-13T02:16:33.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-13T02:16:33.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-13T02:16:33.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-13T02:16:33.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-13T02:16:33.868 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-13T02:16:33.868 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:16:33.868 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:16:33.868 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:16:33.868 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-13T02:16:34.023 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: local last_scrub=2026-02-13T02:14:07.964003+0000 2026-02-13T02:16:34.023 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: pg_deep_scrub: ceph pg deep-scrub 3.0 2026-02-13T02:16:34.175 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2026-02-13T02:16:34.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: pg_deep_scrub: wait_for_scrub 3.0 2026-02-13T02:14:07.964003+0000 last_deep_scrub_stamp 2026-02-13T02:16:34.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-13T02:16:34.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:14:07.964003+0000 2026-02-13T02:16:34.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_deep_scrub_stamp 2026-02-13T02:16:34.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:16:34.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:16:34.186 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-13T02:16:34.186 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:16:34.186 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:16:34.187 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:16:34.187 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-13T02:16:34.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:14:07.964003+0000 '>' 2026-02-13T02:14:07.964003+0000 2026-02-13T02:16:34.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:16:35.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:16:35.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:16:35.347 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-13T02:16:35.347 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:16:35.347 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:16:35.347 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:16:35.348 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-13T02:16:35.505 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:14:07.964003+0000 '>' 2026-02-13T02:14:07.964003+0000 2026-02-13T02:16:35.505 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:16:36.506 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:16:36.506 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:16:36.507 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-13T02:16:36.507 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:16:36.507 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:16:36.508 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:16:36.508 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-13T02:16:36.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:14:07.964003+0000 '>' 2026-02-13T02:14:07.964003+0000 2026-02-13T02:16:36.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:16:37.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:16:37.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:16:37.667 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-13T02:16:37.667 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:16:37.667 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:16:37.668 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:16:37.668 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-13T02:16:37.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:14:07.964003+0000 '>' 2026-02-13T02:14:07.964003+0000 2026-02-13T02:16:37.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:16:38.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:16:38.827 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:16:38.827 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-13T02:16:38.827 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:16:38.827 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:16:38.828 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:16:38.828 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-13T02:16:38.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:16:34.566852+0000 '>' 2026-02-13T02:14:07.964003+0000 2026-02-13T02:16:38.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:16:38.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2210: TEST_corrupt_scrub_replicated: err_strings=() 2026-02-13T02:16:38.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2211: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2026-02-13T02:16:38.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2212: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' 2026-02-13T02:16:38.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2213: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' 2026-02-13T02:16:38.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2214: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' 2026-02-13T02:16:38.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2215: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2026-02-13T02:16:38.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2216: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2026-02-13T02:16:38.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2217: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2026-02-13T02:16:38.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2218: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2026-02-13T02:16:38.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2219: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' 2026-02-13T02:16:38.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2220: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2026-02-13T02:16:38.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2221: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2026-02-13T02:16:38.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2222: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' 2026-02-13T02:16:38.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2223: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' 2026-02-13T02:16:38.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2224: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' 2026-02-13T02:16:38.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2225: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' 2026-02-13T02:16:38.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2226: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2026-02-13T02:16:38.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2227: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2026-02-13T02:16:38.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2228: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' 2026-02-13T02:16:38.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2229: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2026-02-13T02:16:38.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2230: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2026-02-13T02:16:38.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2231: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2026-02-13T02:16:38.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2232: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2026-02-13T02:16:38.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2233: TEST_corrupt_scrub_replicated: err_strings[22]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2026-02-13T02:16:38.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2234: TEST_corrupt_scrub_replicated: err_strings[23]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2026-02-13T02:16:38.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2235: TEST_corrupt_scrub_replicated: err_strings[24]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' 2026-02-13T02:16:38.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2236: TEST_corrupt_scrub_replicated: err_strings[25]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2026-02-13T02:16:38.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2237: TEST_corrupt_scrub_replicated: err_strings[26]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' 2026-02-13T02:16:38.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2238: TEST_corrupt_scrub_replicated: err_strings[27]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' 2026-02-13T02:16:38.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2239: TEST_corrupt_scrub_replicated: err_strings[28]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2026-02-13T02:16:38.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2240: TEST_corrupt_scrub_replicated: err_strings[29]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' 2026-02-13T02:16:38.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2241: TEST_corrupt_scrub_replicated: err_strings[30]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' 2026-02-13T02:16:38.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2242: TEST_corrupt_scrub_replicated: err_strings[31]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' 2026-02-13T02:16:38.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2243: TEST_corrupt_scrub_replicated: err_strings[32]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' 2026-02-13T02:16:38.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2244: TEST_corrupt_scrub_replicated: err_strings[33]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2026-02-13T02:16:38.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2245: TEST_corrupt_scrub_replicated: err_strings[34]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2026-02-13T02:16:38.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2246: TEST_corrupt_scrub_replicated: err_strings[35]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* v=3 cannot decode .* Malformed input' 2026-02-13T02:16:38.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2247: TEST_corrupt_scrub_replicated: err_strings[36]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2026-02-13T02:16:38.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2248: TEST_corrupt_scrub_replicated: err_strings[37]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' 2026-02-13T02:16:38.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2249: TEST_corrupt_scrub_replicated: err_strings[38]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' 2026-02-13T02:16:38.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:38.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:38.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:38.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:38.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:38.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:39.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:39.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:39.015 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:39.015 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:39.023 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:39.023 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:39.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:39.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:39.037 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:39.037 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:39.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:39.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:39.051 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:39.051 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:39.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:39.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:39.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:39.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:39.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:39.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:39.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:39.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:39.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:39.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:39.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:39.099 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:39.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:39.107 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:39.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:39.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:39.123 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:39.123 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:39.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:39.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:39.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:39.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:39.144 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:39.144 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:39.152 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:39.152 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:39.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:39.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:39.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:39.167 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:39.175 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:39.175 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:39.182 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:39.182 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:39.190 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:39.190 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:39.198 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:39.198 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:39.206 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:39.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:39.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:39.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:39.223 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:39.223 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:39.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:39.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:39.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:39.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:39.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:39.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:39.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:39.258 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* v=3 cannot decode .* Malformed input' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:39.267 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:39.267 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:39.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:39.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:39.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-13T02:16:39.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' td/osd-scrub-repair/osd.1.log 2026-02-13T02:16:39.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2260: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2026-02-13T02:16:39.322 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2262: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2026-02-13T02:16:39.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2262: TEST_corrupt_scrub_replicated: test 1 = 1 2026-02-13T02:16:39.338 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2264: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2026-02-13T02:16:39.348 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2264: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2026-02-13T02:16:39.348 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2266: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2026-02-13T02:16:39.366 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2268: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2026-02-13T02:16:39.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2268: TEST_corrupt_scrub_replicated: epoch=203 2026-02-13T02:16:39.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2270: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2026-02-13T02:16:39.377 INFO:tasks.workunit.client.0.trial112.stderr: . as $in 2026-02-13T02:16:39.378 INFO:tasks.workunit.client.0.trial112.stderr: | if type == "object" then 2026-02-13T02:16:39.378 INFO:tasks.workunit.client.0.trial112.stderr: reduce keys[] as $key 2026-02-13T02:16:39.378 INFO:tasks.workunit.client.0.trial112.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-13T02:16:39.378 INFO:tasks.workunit.client.0.trial112.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-13T02:16:39.378 INFO:tasks.workunit.client.0.trial112.stderr: else f 2026-02-13T02:16:39.378 INFO:tasks.workunit.client.0.trial112.stderr: end; 2026-02-13T02:16:39.378 INFO:tasks.workunit.client.0.trial112.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-13T02:16:39.378 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-13T02:16:39.378 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-13T02:16:39.378 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-13T02:16:39.378 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-13T02:16:39.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2270: TEST_corrupt_scrub_replicated: jq .inconsistents 2026-02-13T02:16:39.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2270: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-13T02:16:39.403 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3698: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2026-02-13T02:16:39.403 INFO:tasks.workunit.client.0.trial112.stderr: . as $in 2026-02-13T02:16:39.403 INFO:tasks.workunit.client.0.trial112.stderr: | if type == "object" then 2026-02-13T02:16:39.403 INFO:tasks.workunit.client.0.trial112.stderr: reduce keys[] as $key 2026-02-13T02:16:39.403 INFO:tasks.workunit.client.0.trial112.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-13T02:16:39.403 INFO:tasks.workunit.client.0.trial112.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-13T02:16:39.403 INFO:tasks.workunit.client.0.trial112.stderr: else f 2026-02-13T02:16:39.403 INFO:tasks.workunit.client.0.trial112.stderr: end; 2026-02-13T02:16:39.404 INFO:tasks.workunit.client.0.trial112.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-13T02:16:39.404 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-13T02:16:39.404 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-13T02:16:39.404 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-13T02:16:39.404 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2026-02-13T02:16:39.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3698: TEST_corrupt_scrub_replicated: jq .inconsistents 2026-02-13T02:16:39.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3698: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-13T02:16:39.435 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3699: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-13T02:16:39.435 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2513: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-13T02:16:39.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3700: TEST_corrupt_scrub_replicated: test no = yes 2026-02-13T02:16:39.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3705: TEST_corrupt_scrub_replicated: test '' = yes 2026-02-13T02:16:39.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3710: TEST_corrupt_scrub_replicated: repair 3.0 2026-02-13T02:16:39.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=3.0 2026-02-13T02:16:39.438 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 3.0 2026-02-13T02:16:39.438 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:16:39.438 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:16:39.438 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:16:39.438 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:16:39.598 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-13T02:16:34.566852+0000 2026-02-13T02:16:39.598 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 3.0 2026-02-13T02:16:39.748 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 3.0 on osd.1 to repair 2026-02-13T02:16:39.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 3.0 2026-02-13T02:16:34.566852+0000 2026-02-13T02:16:39.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-13T02:16:39.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:16:34.566852+0000 2026-02-13T02:16:39.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:16:39.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:16:39.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:16:39.759 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-13T02:16:39.759 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:16:39.759 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:16:39.760 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:16:39.760 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:16:39.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:16:34.566852+0000 '>' 2026-02-13T02:16:34.566852+0000 2026-02-13T02:16:39.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:16:40.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:16:40.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:16:40.923 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-13T02:16:40.923 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:16:40.923 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:16:40.923 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:16:40.923 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:16:41.086 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:16:34.566852+0000 '>' 2026-02-13T02:16:34.566852+0000 2026-02-13T02:16:41.086 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:16:42.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:16:42.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:16:42.088 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-13T02:16:42.088 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:16:42.088 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:16:42.088 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:16:42.088 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:16:42.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:16:34.566852+0000 '>' 2026-02-13T02:16:34.566852+0000 2026-02-13T02:16:42.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:16:43.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:16:43.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:16:43.250 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-13T02:16:43.250 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:16:43.250 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:16:43.250 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:16:43.251 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:16:43.410 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:16:34.566852+0000 '>' 2026-02-13T02:16:34.566852+0000 2026-02-13T02:16:43.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:16:44.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:16:44.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:16:44.413 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-13T02:16:44.413 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:16:44.413 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:16:44.413 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:16:44.413 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:16:44.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:16:41.632464+0000 '>' 2026-02-13T02:16:34.566852+0000 2026-02-13T02:16:44.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:16:44.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3711: TEST_corrupt_scrub_replicated: wait_for_clean 2026-02-13T02:16:44.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:16:44.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:16:44.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:16:44.571 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:16:44.571 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:16:44.571 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:16:44.572 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:16:44.572 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:16:44.572 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:16:44.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:16:44.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:16:44.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:16:44.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:16:44.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:16:44.689 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:16:44.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:16:44.850 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:16:44.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:16:44.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:16:44.850 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:16:44.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=850403524617 2026-02-13T02:16:44.924 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 850403524617 2026-02-13T02:16:44.924 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-850403524617' 2026-02-13T02:16:44.924 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:16:44.924 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:16:44.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=871878361095 2026-02-13T02:16:44.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 871878361095 2026-02-13T02:16:44.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-850403524617 1-871878361095' 2026-02-13T02:16:44.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:16:44.998 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-850403524617 2026-02-13T02:16:44.998 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:16:44.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:16:45.000 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-850403524617 2026-02-13T02:16:45.000 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:16:45.001 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=850403524617 2026-02-13T02:16:45.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 850403524617' 2026-02-13T02:16:45.002 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 850403524617 2026-02-13T02:16:45.002 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:16:45.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 850403524616 -lt 850403524617 2026-02-13T02:16:45.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:16:46.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:16:46.167 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:16:46.331 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 850403524616 -lt 850403524617 2026-02-13T02:16:46.332 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:16:47.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:16:47.334 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:16:47.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 850403524617 -lt 850403524617 2026-02-13T02:16:47.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:16:47.493 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-871878361095 2026-02-13T02:16:47.493 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:16:47.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:16:47.496 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-871878361095 2026-02-13T02:16:47.496 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:16:47.498 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=871878361095 2026-02-13T02:16:47.498 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 871878361095' 2026-02-13T02:16:47.498 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 871878361095 2026-02-13T02:16:47.498 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:16:47.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 871878361095 -lt 871878361095 2026-02-13T02:16:47.663 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:16:47.663 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:16:47.664 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:16:47.857 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:16:47.857 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:16:47.858 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:16:47.859 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:16:47.859 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:16:47.859 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:16:47.859 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:16:47.859 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:16:48.015 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:16:48.016 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:16:48.016 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:16:48.017 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:16:48.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:16:48.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:16:48.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:16:48.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3714: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ17 td/osd-scrub-repair/robj17.out 2026-02-13T02:16:48.224 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3715: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ18 td/osd-scrub-repair/robj18.out 2026-02-13T02:16:48.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3717: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2026-02-13T02:16:48.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3718: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2026-02-13T02:16:48.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3719: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2026-02-13T02:16:48.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3720: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2026-02-13T02:16:48.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3722: TEST_corrupt_scrub_replicated: '[' 0 '!=' 0 ']' 2026-02-13T02:16:48.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3728: TEST_corrupt_scrub_replicated: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2026-02-13T02:16:48.448 INFO:tasks.workunit.client.0.trial112.stderr:pool 'csr_pool' removed 2026-02-13T02:16:48.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-13T02:16:48.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:16:48.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:16:48.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:16:48.461 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:16:48.462 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:16:48.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:16:48.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:16:48.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:16:48.773 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:16:48.774 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:16:48.775 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:16:48.776 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:16:48.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:16:48.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:16:48.779 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:16:48.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:16:48.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:16:48.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:16:48.781 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:16:48.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:16:48.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:16:48.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:16:48.816 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:16:48.816 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:16:48.816 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:16:48.816 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:16:48.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:16:48.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:16:48.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-13T02:16:48.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-13T02:16:48.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-13T02:16:48.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-13T02:16:48.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:16:48.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:16:48.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:16:48.818 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:16:48.818 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:16:48.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:16:48.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:16:48.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:16:48.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:16:48.822 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:16:48.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:16:48.823 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:16:48.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:16:48.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:16:48.825 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:16:48.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:16:48.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:16:48.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:16:48.826 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:16:48.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:16:48.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:16:48.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:16:48.831 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:16:48.831 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:16:48.831 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:16:48.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:16:48.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:16:48.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:16:48.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-13T02:16:48.834 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T02:16:48.834 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:16:48.834 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:16:48.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.35207 2026-02-13T02:16:48.836 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T02:16:48.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-13T02:16:48.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T02:16:48.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-13T02:16:48.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_snapset_scrub_rep td/osd-scrub-repair 2026-02-13T02:16:48.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6100: TEST_corrupt_snapset_scrub_rep: local dir=td/osd-scrub-repair 2026-02-13T02:16:48.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6101: TEST_corrupt_snapset_scrub_rep: local poolname=csr_pool 2026-02-13T02:16:48.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6102: TEST_corrupt_snapset_scrub_rep: local total_objs=2 2026-02-13T02:16:48.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6104: TEST_corrupt_snapset_scrub_rep: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2026-02-13T02:16:48.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-13T02:16:48.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-13T02:16:48.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-13T02:16:48.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-13T02:16:48.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-13T02:16:48.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2026-02-13T02:16:48.859 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-13T02:16:48.859 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:16:48.859 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:16:48.860 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:16:48.860 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:16:48.860 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:16:48.860 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:16:48.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2026-02-13T02:16:48.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-13T02:16:48.891 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-13T02:16:48.891 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:16:48.891 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:16:48.891 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-13T02:16:48.892 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-13T02:16:48.892 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:16:48.892 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:16:48.892 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:16:48.893 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:16:48.893 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:16:48.893 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:16:48.893 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:16:48.893 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:16:48.893 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get fsid 2026-02-13T02:16:48.950 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-13T02:16:48.950 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:16:48.950 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:16:48.951 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-13T02:16:48.951 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-13T02:16:48.952 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:16:48.952 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:16:48.952 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:16:48.952 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:16:48.952 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:16:48.952 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:16:48.952 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:16:48.953 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:16:48.953 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get mon_host 2026-02-13T02:16:49.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6105: TEST_corrupt_snapset_scrub_rep: run_mgr td/osd-scrub-repair x 2026-02-13T02:16:49.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-13T02:16:49.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-13T02:16:49.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-13T02:16:49.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-13T02:16:49.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-13T02:16:49.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-13T02:16:49.105 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-13T02:16:49.105 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:16:49.105 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:16:49.106 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:16:49.106 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:16:49.106 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:16:49.106 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:16:49.107 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:16:49.108 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:16:49.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6106: TEST_corrupt_snapset_scrub_rep: run_osd td/osd-scrub-repair 0 2026-02-13T02:16:49.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:16:49.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:16:49.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-13T02:16:49.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:16:49.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:16:49.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:16:49.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:16:49.128 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:16:49.128 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:16:49.128 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:16:49.128 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:16:49.128 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:16:49.128 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:16:49.128 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:16:49.128 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:16:49.128 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:16:49.128 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:16:49.128 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:16:49.128 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:16:49.129 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:16:49.129 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:16:49.129 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:16:49.129 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:16:49.129 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:16:49.129 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:16:49.129 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:16:49.129 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:16:49.129 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:16:49.129 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:16:49.129 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:16:49.129 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:16:49.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:16:49.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:16:49.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:16:49.130 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:16:49.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7275f752-62c4-4859-8272-cbb9b227e3c2 2026-02-13T02:16:49.131 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 7275f752-62c4-4859-8272-cbb9b227e3c2 2026-02-13T02:16:49.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 7275f752-62c4-4859-8272-cbb9b227e3c2' 2026-02-13T02:16:49.131 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:16:49.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCRiY5pfS7GBxAAcNj//y5sWNqiNOqbVCqXiQ== 2026-02-13T02:16:49.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCRiY5pfS7GBxAAcNj//y5sWNqiNOqbVCqXiQ=="}' 2026-02-13T02:16:49.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7275f752-62c4-4859-8272-cbb9b227e3c2 -i td/osd-scrub-repair/0/new.json 2026-02-13T02:16:49.231 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:16:49.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-13T02:16:49.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCRiY5pfS7GBxAAcNj//y5sWNqiNOqbVCqXiQ== --osd-uuid 7275f752-62c4-4859-8272-cbb9b227e3c2 2026-02-13T02:16:49.255 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:16:49.242+0000 7f210ac0aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:16:49.257 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:16:49.246+0000 7f210ac0aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:16:49.257 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:16:49.246+0000 7f210ac0aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:16:49.258 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:16:49.246+0000 7f210ac0aa40 -1 bdev(0x55da3a10ec00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-13T02:16:49.258 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:16:49.246+0000 7f210ac0aa40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-13T02:16:49.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-13T02:16:49.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:16:49.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-13T02:16:49.994 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T02:16:49.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:16:50.102 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T02:16:50.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-13T02:16:50.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:16:50.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:16:50.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:16:50.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:16:50.125 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:16:50.114+0000 7f7563194a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:16:50.127 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:16:50.114+0000 7f7563194a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:16:50.128 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:16:50.114+0000 7f7563194a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:16:50.199 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-13T02:16:50.199 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:16:50.199 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:16:50.199 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:16:50.199 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:16:50.199 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:16:50.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:16:50.200 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:16:50.201 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:16:50.201 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:16:50.291 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:16:50.647 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:16:50.634+0000 7f7563194a40 -1 Falling back to public interface 2026-02-13T02:16:51.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:16:51.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:16:51.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:16:51.294 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:16:51.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:16:51.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:16:51.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:16:51.539 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:16:51.526+0000 7f7563194a40 -1 osd.0 0 log_to_monitors true 2026-02-13T02:16:52.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:16:52.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:16:52.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:16:52.392 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:16:52.393 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:16:52.393 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:16:52.553 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3661762313,v1:127.0.0.1:6801/3661762313] [v2:127.0.0.1:6802/3661762313,v1:127.0.0.1:6803/3661762313] exists,up 7275f752-62c4-4859-8272-cbb9b227e3c2 2026-02-13T02:16:52.558 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:16:52.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:16:52.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:16:52.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6107: TEST_corrupt_snapset_scrub_rep: run_osd td/osd-scrub-repair 1 2026-02-13T02:16:52.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:16:52.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:16:52.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-13T02:16:52.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:16:52.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:16:52.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:16:52.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:16:52.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:16:52.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:16:52.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:16:52.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:16:52.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:16:52.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:16:52.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:16:52.561 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:16:52.561 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:16:52.561 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:16:52.561 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:16:52.561 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:16:52.561 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:16:52.561 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:16:52.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:16:52.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:16:52.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:16:52.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:16:52.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:16:52.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:16:52.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:16:52.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:16:52.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:16:52.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:16:52.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:16:52.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:16:52.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:16:52.562 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:16:52.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=69b5b2c2-ca43-4c66-9c1e-0aab71d9a11f 2026-02-13T02:16:52.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 69b5b2c2-ca43-4c66-9c1e-0aab71d9a11f' 2026-02-13T02:16:52.563 INFO:tasks.workunit.client.0.trial112.stdout:add osd1 69b5b2c2-ca43-4c66-9c1e-0aab71d9a11f 2026-02-13T02:16:52.563 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:16:52.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCUiY5pP9VmIRAA3T7WlE9E2xJgA7jELfI+hA== 2026-02-13T02:16:52.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCUiY5pP9VmIRAA3T7WlE9E2xJgA7jELfI+hA=="}' 2026-02-13T02:16:52.572 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 69b5b2c2-ca43-4c66-9c1e-0aab71d9a11f -i td/osd-scrub-repair/1/new.json 2026-02-13T02:16:52.739 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:16:52.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-13T02:16:52.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCUiY5pP9VmIRAA3T7WlE9E2xJgA7jELfI+hA== --osd-uuid 69b5b2c2-ca43-4c66-9c1e-0aab71d9a11f 2026-02-13T02:16:52.767 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:16:52.754+0000 7efe4ebc9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:16:52.768 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:16:52.754+0000 7efe4ebc9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:16:52.769 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:16:52.758+0000 7efe4ebc9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:16:52.770 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:16:52.758+0000 7efe4ebc9a40 -1 bdev(0x5571991b0c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-13T02:16:52.770 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:16:52.758+0000 7efe4ebc9a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-13T02:16:53.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-13T02:16:53.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:16:53.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-13T02:16:53.449 INFO:tasks.workunit.client.0.trial112.stdout:adding osd1 key to auth repository 2026-02-13T02:16:53.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:16:53.660 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T02:16:53.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-13T02:16:53.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:16:53.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:16:53.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:16:53.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:16:53.675 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:16:53.662+0000 7f1aa8271a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:16:53.677 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:16:53.666+0000 7f1aa8271a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:16:53.677 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:16:53.666+0000 7f1aa8271a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:16:53.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-13T02:16:53.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:16:53.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:16:53.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:16:53.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:16:53.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:16:53.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:16:53.816 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:16:53.816 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:16:53.816 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:16:53.980 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:16:54.058 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:16:54.046+0000 7f1aa8271a40 -1 Falling back to public interface 2026-02-13T02:16:54.679 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:16:54.666+0000 7f1aa8271a40 -1 osd.1 0 log_to_monitors true 2026-02-13T02:16:54.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:16:54.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:16:54.982 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:16:54.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:16:54.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:16:54.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:16:55.152 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:16:56.154 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:16:56.154 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:16:56.154 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:16:56.155 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:16:56.155 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:16:56.155 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:16:56.319 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3532159507,v1:127.0.0.1:6811/3532159507] [v2:127.0.0.1:6812/3532159507,v1:127.0.0.1:6813/3532159507] exists,up 69b5b2c2-ca43-4c66-9c1e-0aab71d9a11f 2026-02-13T02:16:56.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:16:56.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:16:56.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:16:56.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6108: TEST_corrupt_snapset_scrub_rep: create_rbd_pool 2026-02-13T02:16:56.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-13T02:16:56.471 INFO:tasks.workunit.client.0.trial112.stderr:pool 'rbd' does not exist 2026-02-13T02:16:56.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-13T02:16:56.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-13T02:16:56.701 INFO:tasks.workunit.client.0.trial112.stderr:pool 'rbd' created 2026-02-13T02:16:56.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:16:57.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-13T02:16:58.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6109: TEST_corrupt_snapset_scrub_rep: wait_for_clean 2026-02-13T02:16:58.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:16:58.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:16:58.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:16:58.043 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:16:58.044 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:16:58.045 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:16:58.045 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:16:58.045 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:16:58.045 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:16:58.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:16:58.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:16:58.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:16:58.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:16:58.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:16:58.163 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:16:58.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:16:58.324 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:16:58.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:16:58.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:16:58.325 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:16:58.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-13T02:16:58.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-13T02:16:58.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-13T02:16:58.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:16:58.399 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:16:58.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705666 2026-02-13T02:16:58.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705666 2026-02-13T02:16:58.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705666' 2026-02-13T02:16:58.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:16:58.476 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-13T02:16:58.476 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:16:58.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:16:58.478 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-13T02:16:58.478 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:16:58.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-13T02:16:58.480 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836483 2026-02-13T02:16:58.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-13T02:16:58.481 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:16:58.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-13T02:16:58.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:16:59.642 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:16:59.642 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:16:59.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-13T02:16:59.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:16:59.804 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705666 2026-02-13T02:16:59.804 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:16:59.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:16:59.806 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705666 2026-02-13T02:16:59.806 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:16:59.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705666 2026-02-13T02:16:59.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705666' 2026-02-13T02:16:59.808 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705666 2026-02-13T02:16:59.809 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:16:59.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705666 -lt 38654705666 2026-02-13T02:16:59.982 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:16:59.983 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:16:59.983 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:17:00.181 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-13T02:17:00.182 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:17:00.182 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:17:00.182 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:17:00.182 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:17:00.182 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:17:00.182 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:17:00.182 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:17:00.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-13T02:17:00.345 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:17:00.345 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:17:00.346 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:17:00.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-13T02:17:00.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:17:00.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:17:00.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6111: TEST_corrupt_snapset_scrub_rep: create_pool foo 1 2026-02-13T02:17:00.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 1 2026-02-13T02:17:00.747 INFO:tasks.workunit.client.0.trial112.stderr:pool 'foo' created 2026-02-13T02:17:00.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:17:01.760 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6112: TEST_corrupt_snapset_scrub_rep: create_pool csr_pool 1 1 2026-02-13T02:17:01.760 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create csr_pool 1 1 2026-02-13T02:17:01.970 INFO:tasks.workunit.client.0.trial112.stderr:pool 'csr_pool' created 2026-02-13T02:17:01.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:17:02.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6113: TEST_corrupt_snapset_scrub_rep: wait_for_clean 2026-02-13T02:17:02.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:17:02.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:17:02.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:17:02.985 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:17:02.986 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:17:02.986 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:17:02.986 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:17:02.986 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:17:02.986 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:17:03.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:17:03.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:17:03.107 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:17:03.107 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:17:03.107 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:17:03.107 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:17:03.277 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:17:03.277 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:17:03.277 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:17:03.277 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:17:03.278 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:17:03.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836485 2026-02-13T02:17:03.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836485 2026-02-13T02:17:03.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485' 2026-02-13T02:17:03.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:17:03.352 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:17:03.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705668 2026-02-13T02:17:03.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705668 2026-02-13T02:17:03.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-38654705668' 2026-02-13T02:17:03.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:17:03.426 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836485 2026-02-13T02:17:03.426 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:17:03.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:17:03.429 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836485 2026-02-13T02:17:03.429 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:17:03.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836485 2026-02-13T02:17:03.431 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836485 2026-02-13T02:17:03.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-13T02:17:03.431 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:17:03.599 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 21474836485 2026-02-13T02:17:03.599 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:17:03.600 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705668 2026-02-13T02:17:03.600 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:17:03.601 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:17:03.602 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705668 2026-02-13T02:17:03.603 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:17:03.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705668 2026-02-13T02:17:03.604 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705668 2026-02-13T02:17:03.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705668' 2026-02-13T02:17:03.605 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:17:03.768 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705667 -lt 38654705668 2026-02-13T02:17:03.768 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:17:04.769 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:17:04.770 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:17:04.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705667 -lt 38654705668 2026-02-13T02:17:04.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:17:05.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:17:05.940 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:17:06.108 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705668 -lt 38654705668 2026-02-13T02:17:06.108 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:17:06.109 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:17:06.109 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:17:06.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:17:06.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:17:06.307 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:17:06.307 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:17:06.307 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:17:06.307 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:17:06.307 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:17:06.307 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:17:06.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:17:06.465 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:17:06.465 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:17:06.465 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:17:06.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:17:06.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:17:06.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:17:06.656 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6115: TEST_corrupt_snapset_scrub_rep: seq 1 2 2026-02-13T02:17:06.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6115: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2026-02-13T02:17:06.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6116: TEST_corrupt_snapset_scrub_rep: objname=ROBJ1 2026-02-13T02:17:06.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6117: TEST_corrupt_snapset_scrub_rep: add_something td/osd-scrub-repair csr_pool ROBJ1 2026-02-13T02:17:06.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:17:06.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-13T02:17:06.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ1 2026-02-13T02:17:06.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:17:06.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:17:06.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:17:06.858 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:17:06.871 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:17:07.064 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:17:07.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:17:07.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:17:07.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:17:07.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6119: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2026-02-13T02:17:07.122 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6120: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2026-02-13T02:17:07.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6115: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2026-02-13T02:17:07.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6116: TEST_corrupt_snapset_scrub_rep: objname=ROBJ2 2026-02-13T02:17:07.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6117: TEST_corrupt_snapset_scrub_rep: add_something td/osd-scrub-repair csr_pool ROBJ2 2026-02-13T02:17:07.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:17:07.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-13T02:17:07.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ2 2026-02-13T02:17:07.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:17:07.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:17:07.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:17:07.366 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:17:07.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:17:07.561 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:17:07.574 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:17:07.574 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:17:07.574 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:17:07.601 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6119: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2026-02-13T02:17:07.626 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6120: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2026-02-13T02:17:07.654 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6123: TEST_corrupt_snapset_scrub_rep: get_pg csr_pool ROBJ0 2026-02-13T02:17:07.654 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=csr_pool 2026-02-13T02:17:07.654 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=ROBJ0 2026-02-13T02:17:07.655 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map csr_pool ROBJ0 2026-02-13T02:17:07.655 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T02:17:07.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6123: TEST_corrupt_snapset_scrub_rep: local pg=3.0 2026-02-13T02:17:07.821 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6124: TEST_corrupt_snapset_scrub_rep: get_primary csr_pool ROBJ0 2026-02-13T02:17:07.821 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=csr_pool 2026-02-13T02:17:07.821 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=ROBJ0 2026-02-13T02:17:07.822 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map csr_pool ROBJ0 2026-02-13T02:17:07.822 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T02:17:07.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6124: TEST_corrupt_snapset_scrub_rep: local primary=1 2026-02-13T02:17:07.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6126: TEST_corrupt_snapset_scrub_rep: rados -p csr_pool mksnap snap1 2026-02-13T02:17:08.053 INFO:tasks.workunit.client.0.trial112.stdout:created pool csr_pool snap snap1 2026-02-13T02:17:08.055 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6127: TEST_corrupt_snapset_scrub_rep: echo -n head_of_snapshot_data 2026-02-13T02:17:08.055 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6129: TEST_corrupt_snapset_scrub_rep: seq 1 2 2026-02-13T02:17:08.057 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6129: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2026-02-13T02:17:08.057 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6130: TEST_corrupt_snapset_scrub_rep: objname=ROBJ1 2026-02-13T02:17:08.057 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6133: TEST_corrupt_snapset_scrub_rep: expr 1 % 2 2026-02-13T02:17:08.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6133: TEST_corrupt_snapset_scrub_rep: local osd=1 2026-02-13T02:17:08.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6135: TEST_corrupt_snapset_scrub_rep: case $i in 2026-02-13T02:17:08.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6137: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/change 2026-02-13T02:17:08.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6138: TEST_corrupt_snapset_scrub_rep: objectstore_tool td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2026-02-13T02:17:08.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:17:08.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:17:08.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:17:08.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:17:08.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2026-02-13T02:17:08.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:17:08.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:17:08.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:17:08.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:17:08.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:17:08.081 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:17:08.081 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:17:08.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:17:08.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:17:08.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:17:08.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:17:08.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2026-02-13T02:17:08.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:17:08.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:17:08.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:17:08.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:17:08.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:17:08.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 --head ROBJ1 clear-snapset corrupt 2026-02-13T02:17:08.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-13T02:17:08.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:17:08.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:17:08.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:17:08.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:17:08.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:17:08.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:17:08.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:17:08.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:17:08.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:17:08.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:17:08.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:17:08.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:17:08.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:17:08.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:17:08.688 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:17:08.688 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:17:08.688 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:17:08.688 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:17:08.688 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:17:08.689 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:17:08.689 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:17:08.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:17:08.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:17:08.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:17:08.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:17:08.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:17:08.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:17:08.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:17:08.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:17:08.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:17:08.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:17:08.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:17:08.692 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:17:08.692 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:17:08.692 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:17:08.692 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:17:08.692 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:17:08.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:17:08.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:17:08.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:17:08.707 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:17:08.694+0000 7f4dd1386a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:17:08.708 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:17:08.698+0000 7f4dd1386a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:17:08.709 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:17:08.698+0000 7f4dd1386a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:17:08.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:17:08.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:17:08.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:17:08.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:17:08.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:17:08.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:17:08.852 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:17:08.852 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:17:08.852 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:17:08.852 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:17:09.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:17:09.095 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:17:09.082+0000 7f4dd1386a40 -1 Falling back to public interface 2026-02-13T02:17:09.632 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:17:09.618+0000 7f4dd1386a40 -1 osd.1 28 log_to_monitors true 2026-02-13T02:17:10.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:17:10.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:17:10.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:17:10.014 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:17:10.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:17:10.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:17:10.175 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 32 up_thru 32 down_at 29 last_clean_interval [9,28) [v2:127.0.0.1:6810/3886076291,v1:127.0.0.1:6811/3886076291] [v2:127.0.0.1:6812/3886076291,v1:127.0.0.1:6813/3886076291] exists,up 69b5b2c2-ca43-4c66-9c1e-0aab71d9a11f 2026-02-13T02:17:10.175 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:17:10.175 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:17:10.176 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:17:10.176 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:17:10.176 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:17:10.176 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:17:10.176 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:17:10.177 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:17:10.177 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:17:10.177 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:17:10.177 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:17:10.177 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:17:10.177 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:17:10.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:17:10.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:17:10.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:17:10.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:17:10.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:17:10.292 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:17:10.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:17:10.450 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:17:10.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:17:10.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:17:10.451 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:17:10.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836487 2026-02-13T02:17:10.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836487 2026-02-13T02:17:10.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487' 2026-02-13T02:17:10.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:17:10.525 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:17:10.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953474 2026-02-13T02:17:10.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953474 2026-02-13T02:17:10.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487 1-137438953474' 2026-02-13T02:17:10.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:17:10.609 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836487 2026-02-13T02:17:10.609 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:17:10.610 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:17:10.611 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836487 2026-02-13T02:17:10.611 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:17:10.613 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836487 2026-02-13T02:17:10.613 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2026-02-13T02:17:10.613 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 21474836487 2026-02-13T02:17:10.613 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:17:10.773 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836487 2026-02-13T02:17:10.773 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:17:11.774 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:17:11.775 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:17:11.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836487 2026-02-13T02:17:11.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:17:11.937 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-137438953474 2026-02-13T02:17:11.937 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:17:11.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:17:11.940 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-137438953474 2026-02-13T02:17:11.940 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:17:11.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953474 2026-02-13T02:17:11.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 137438953474' 2026-02-13T02:17:11.942 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 137438953474 2026-02-13T02:17:11.942 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:17:12.104 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953474 -lt 137438953474 2026-02-13T02:17:12.105 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:17:12.105 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:17:12.105 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:17:12.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:17:12.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:17:12.305 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:17:12.305 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:17:12.305 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:17:12.305 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:17:12.306 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:17:12.306 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:17:12.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:17:12.463 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:17:12.463 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:17:12.463 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:17:12.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:17:12.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:17:12.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:17:12.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6129: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2026-02-13T02:17:12.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6130: TEST_corrupt_snapset_scrub_rep: objname=ROBJ2 2026-02-13T02:17:12.656 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6133: TEST_corrupt_snapset_scrub_rep: expr 2 % 2 2026-02-13T02:17:12.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6133: TEST_corrupt_snapset_scrub_rep: local osd=0 2026-02-13T02:17:12.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6135: TEST_corrupt_snapset_scrub_rep: case $i in 2026-02-13T02:17:12.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6142: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/change 2026-02-13T02:17:12.678 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6143: TEST_corrupt_snapset_scrub_rep: objectstore_tool td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2026-02-13T02:17:12.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:17:12.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:17:12.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T02:17:12.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:17:12.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2026-02-13T02:17:12.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:17:12.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:17:12.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T02:17:12.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:17:12.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T02:17:12.679 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:17:12.680 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:17:12.680 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:17:12.680 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:17:12.680 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:17:12.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:17:12.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2026-02-13T02:17:12.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:17:12.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:17:12.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:17:12.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:17:12.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:17:12.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 --head ROBJ2 clear-snapset corrupt 2026-02-13T02:17:13.271 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-13T02:17:13.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:17:13.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:17:13.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:17:13.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:17:13.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:17:13.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:17:13.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:17:13.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:17:13.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:17:13.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:17:13.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:17:13.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:17:13.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:17:13.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:17:13.273 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:17:13.273 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:17:13.273 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:17:13.275 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:17:13.275 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:17:13.276 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:17:13.276 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:17:13.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:17:13.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:17:13.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:17:13.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:17:13.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:17:13.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:17:13.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:17:13.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:17:13.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:17:13.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:17:13.277 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:17:13.277 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:17:13.277 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T02:17:13.277 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:17:13.277 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T02:17:13.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:17:13.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:17:13.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:17:13.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:17:13.298 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:17:13.286+0000 7f46ace4fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:17:13.300 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:17:13.286+0000 7f46ace4fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:17:13.301 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:17:13.290+0000 7f46ace4fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:17:13.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:17:13.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:17:13.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:17:13.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:17:13.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:17:13.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:17:13.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:17:13.434 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:17:13.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:17:13.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:17:13.601 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:17:13.714 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:17:13.702+0000 7f46ace4fa40 -1 Falling back to public interface 2026-02-13T02:17:14.292 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:17:14.278+0000 7f46ace4fa40 -1 osd.0 33 log_to_monitors true 2026-02-13T02:17:14.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:17:14.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:17:14.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:17:14.603 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:17:14.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:17:14.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:17:14.766 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 37 up_thru 37 down_at 34 last_clean_interval [5,33) [v2:127.0.0.1:6800/1309594379,v1:127.0.0.1:6801/1309594379] [v2:127.0.0.1:6802/1309594379,v1:127.0.0.1:6803/1309594379] exists,up 7275f752-62c4-4859-8272-cbb9b227e3c2 2026-02-13T02:17:14.767 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:17:14.767 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:17:14.767 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:17:14.767 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:17:14.767 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:17:14.767 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:17:14.767 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:17:14.767 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:17:14.768 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:17:14.768 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:17:14.768 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:17:14.768 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:17:14.769 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:17:14.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:17:14.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:17:14.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:17:14.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:17:14.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:17:14.884 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:17:15.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:17:15.046 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:17:15.047 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:17:15.047 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:17:15.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:17:15.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789954 2026-02-13T02:17:15.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789954 2026-02-13T02:17:15.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-158913789954' 2026-02-13T02:17:15.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:17:15.121 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:17:15.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953476 2026-02-13T02:17:15.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953476 2026-02-13T02:17:15.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-158913789954 1-137438953476' 2026-02-13T02:17:15.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:17:15.193 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-158913789954 2026-02-13T02:17:15.193 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:17:15.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:17:15.195 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-158913789954 2026-02-13T02:17:15.195 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:17:15.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789954 2026-02-13T02:17:15.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 158913789954' 2026-02-13T02:17:15.197 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 158913789954 2026-02-13T02:17:15.197 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:17:15.361 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 158913789954 2026-02-13T02:17:15.361 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:17:16.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:17:16.363 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:17:16.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789954 -lt 158913789954 2026-02-13T02:17:16.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:17:16.531 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-137438953476 2026-02-13T02:17:16.531 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:17:16.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:17:16.533 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-137438953476 2026-02-13T02:17:16.533 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:17:16.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953476 2026-02-13T02:17:16.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 137438953476' 2026-02-13T02:17:16.535 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 137438953476 2026-02-13T02:17:16.536 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:17:16.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953476 -lt 137438953476 2026-02-13T02:17:16.707 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:17:16.707 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:17:16.707 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:17:16.901 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:17:16.901 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:17:16.902 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:17:16.902 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:17:16.902 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:17:16.902 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:17:16.902 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:17:16.903 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:17:17.058 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:17:17.058 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:17:17.059 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:17:17.059 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:17:17.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:17:17.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:17:17.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:17:17.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6148: TEST_corrupt_snapset_scrub_rep: rm td/osd-scrub-repair/change 2026-02-13T02:17:17.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6150: TEST_corrupt_snapset_scrub_rep: pg_scrub 3.0 2026-02-13T02:17:17.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=3.0 2026-02-13T02:17:17.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 3.0 2026-02-13T02:17:17.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-13T02:17:17.250 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-13T02:17:17.250 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:17:17.250 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:17:17.251 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:17:17.251 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:17:17.251 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:17:17.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-13T02:17:17.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-13T02:17:17.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-13T02:17:17.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-13T02:17:17.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:17:17.563 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:17:17.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:17:17.728 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:17:17.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:17:17.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:17:17.729 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:17:17.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789955 2026-02-13T02:17:17.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789955 2026-02-13T02:17:17.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-158913789955' 2026-02-13T02:17:17.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:17:17.803 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:17:17.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953477 2026-02-13T02:17:17.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953477 2026-02-13T02:17:17.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-158913789955 1-137438953477' 2026-02-13T02:17:17.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:17:17.876 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-158913789955 2026-02-13T02:17:17.876 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:17:17.877 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:17:17.878 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-158913789955 2026-02-13T02:17:17.878 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:17:17.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789955 2026-02-13T02:17:17.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 158913789955' 2026-02-13T02:17:17.880 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 158913789955 2026-02-13T02:17:17.880 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:17:18.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789954 -lt 158913789955 2026-02-13T02:17:18.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:17:19.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:17:19.045 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:17:19.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789954 -lt 158913789955 2026-02-13T02:17:19.210 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:17:20.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:17:20.212 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:17:20.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789955 -lt 158913789955 2026-02-13T02:17:20.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:17:20.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-137438953477 2026-02-13T02:17:20.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:17:20.376 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:17:20.376 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-137438953477 2026-02-13T02:17:20.377 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:17:20.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953477 2026-02-13T02:17:20.379 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 137438953477 2026-02-13T02:17:20.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 137438953477' 2026-02-13T02:17:20.379 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:17:20.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953477 -lt 137438953477 2026-02-13T02:17:20.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-13T02:17:20.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-13T02:17:20.542 INFO:tasks.workunit.client.0.trial112.stdout:#---------- 3.0 loop 0 2026-02-13T02:17:20.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-13T02:17:20.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-13T02:17:20.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-13T02:17:20.543 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-13T02:17:20.543 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-13T02:17:20.618 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-13T02:17:20.618 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-13T02:17:20.618 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-13T02:17:20.618 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-13T02:17:20.618 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: get_last_scrub_stamp 3.0 2026-02-13T02:17:20.619 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:17:20.619 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:17:20.619 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:17:20.619 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:17:20.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: local last_scrub=2026-02-13T02:17:01.959665+0000 2026-02-13T02:17:20.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_scrub: ceph pg scrub 3.0 2026-02-13T02:17:20.930 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 3.0 on osd.1 to scrub 2026-02-13T02:17:20.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_scrub: wait_for_scrub 3.0 2026-02-13T02:17:01.959665+0000 2026-02-13T02:17:20.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-13T02:17:20.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:17:01.959665+0000 2026-02-13T02:17:20.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:17:20.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:17:20.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:17:20.941 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-13T02:17:20.941 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:17:20.941 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:17:20.941 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:17:20.941 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:17:21.105 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:17:01.959665+0000 '>' 2026-02-13T02:17:01.959665+0000 2026-02-13T02:17:21.105 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:17:22.107 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:17:22.107 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:17:22.108 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-13T02:17:22.108 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:17:22.108 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:17:22.108 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:17:22.109 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:17:22.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:17:01.959665+0000 '>' 2026-02-13T02:17:01.959665+0000 2026-02-13T02:17:22.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:17:23.266 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:17:23.266 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:17:23.267 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-13T02:17:23.267 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:17:23.267 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:17:23.268 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:17:23.268 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:17:23.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:17:01.959665+0000 '>' 2026-02-13T02:17:01.959665+0000 2026-02-13T02:17:23.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:17:24.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:17:24.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:17:24.432 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-13T02:17:24.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:17:24.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:17:24.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:17:24.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:17:24.599 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:17:01.959665+0000 '>' 2026-02-13T02:17:01.959665+0000 2026-02-13T02:17:24.599 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:17:25.600 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:17:25.600 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:17:25.601 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-13T02:17:25.601 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:17:25.601 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:17:25.601 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:17:25.601 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:17:25.760 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:17:21.485084+0000 '>' 2026-02-13T02:17:01.959665+0000 2026-02-13T02:17:25.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:17:25.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6152: TEST_corrupt_snapset_scrub_rep: rados list-inconsistent-pg csr_pool 2026-02-13T02:17:25.781 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6154: TEST_corrupt_snapset_scrub_rep: jq '. | length' td/osd-scrub-repair/json 2026-02-13T02:17:25.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6154: TEST_corrupt_snapset_scrub_rep: test 1 = 1 2026-02-13T02:17:25.796 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6156: TEST_corrupt_snapset_scrub_rep: jq -r '.[0]' td/osd-scrub-repair/json 2026-02-13T02:17:25.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6156: TEST_corrupt_snapset_scrub_rep: test 3.0 = 3.0 2026-02-13T02:17:25.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6158: TEST_corrupt_snapset_scrub_rep: rados list-inconsistent-obj 3.0 2026-02-13T02:17:25.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6160: TEST_corrupt_snapset_scrub_rep: jq 'def walk(f): 2026-02-13T02:17:25.824 INFO:tasks.workunit.client.0.trial112.stderr: . as $in 2026-02-13T02:17:25.825 INFO:tasks.workunit.client.0.trial112.stderr: | if type == "object" then 2026-02-13T02:17:25.825 INFO:tasks.workunit.client.0.trial112.stderr: reduce keys[] as $key 2026-02-13T02:17:25.825 INFO:tasks.workunit.client.0.trial112.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-13T02:17:25.825 INFO:tasks.workunit.client.0.trial112.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-13T02:17:25.825 INFO:tasks.workunit.client.0.trial112.stderr: else f 2026-02-13T02:17:25.825 INFO:tasks.workunit.client.0.trial112.stderr: end; 2026-02-13T02:17:25.825 INFO:tasks.workunit.client.0.trial112.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-13T02:17:25.825 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-13T02:17:25.825 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-13T02:17:25.825 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-13T02:17:25.825 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-13T02:17:25.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6160: TEST_corrupt_snapset_scrub_rep: jq .inconsistents 2026-02-13T02:17:25.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6160: TEST_corrupt_snapset_scrub_rep: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-13T02:17:25.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6329: TEST_corrupt_snapset_scrub_rep: jq 'def walk(f): 2026-02-13T02:17:25.839 INFO:tasks.workunit.client.0.trial112.stderr: . as $in 2026-02-13T02:17:25.839 INFO:tasks.workunit.client.0.trial112.stderr: | if type == "object" then 2026-02-13T02:17:25.839 INFO:tasks.workunit.client.0.trial112.stderr: reduce keys[] as $key 2026-02-13T02:17:25.839 INFO:tasks.workunit.client.0.trial112.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-13T02:17:25.839 INFO:tasks.workunit.client.0.trial112.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-13T02:17:25.839 INFO:tasks.workunit.client.0.trial112.stderr: else f 2026-02-13T02:17:25.839 INFO:tasks.workunit.client.0.trial112.stderr: end; 2026-02-13T02:17:25.839 INFO:tasks.workunit.client.0.trial112.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-13T02:17:25.839 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-13T02:17:25.839 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-13T02:17:25.839 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-13T02:17:25.840 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2026-02-13T02:17:25.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6329: TEST_corrupt_snapset_scrub_rep: jq .inconsistents 2026-02-13T02:17:25.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6329: TEST_corrupt_snapset_scrub_rep: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-13T02:17:25.852 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6330: TEST_corrupt_snapset_scrub_rep: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-13T02:17:25.852 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2513: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-13T02:17:25.854 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6331: TEST_corrupt_snapset_scrub_rep: test no = yes 2026-02-13T02:17:25.854 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6336: TEST_corrupt_snapset_scrub_rep: test '' = yes 2026-02-13T02:17:25.854 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6341: TEST_corrupt_snapset_scrub_rep: ERRORS=0 2026-02-13T02:17:25.854 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6342: TEST_corrupt_snapset_scrub_rep: declare -a err_strings 2026-02-13T02:17:25.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6343: TEST_corrupt_snapset_scrub_rep: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ1:head : snapset inconsistent' 2026-02-13T02:17:25.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6344: TEST_corrupt_snapset_scrub_rep: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ2:head : snapset inconsistent' 2026-02-13T02:17:25.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6345: TEST_corrupt_snapset_scrub_rep: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 [0-9]*:.*:::ROBJ1:1 : is an unexpected clone' 2026-02-13T02:17:25.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6346: TEST_corrupt_snapset_scrub_rep: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2026-02-13T02:17:25.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6347: TEST_corrupt_snapset_scrub_rep: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 0 missing, 2 inconsistent objects' 2026-02-13T02:17:25.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6348: TEST_corrupt_snapset_scrub_rep: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 4 errors' 2026-02-13T02:17:25.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6350: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2026-02-13T02:17:25.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6352: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ1:head : snapset inconsistent' td/osd-scrub-repair/osd.1.log 2026-02-13T02:17:25.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6350: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2026-02-13T02:17:25.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6352: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ2:head : snapset inconsistent' td/osd-scrub-repair/osd.1.log 2026-02-13T02:17:25.857 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6350: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2026-02-13T02:17:25.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6352: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 [0-9]*:.*:::ROBJ1:1 : is an unexpected clone' td/osd-scrub-repair/osd.1.log 2026-02-13T02:17:25.859 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6350: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2026-02-13T02:17:25.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6352: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2026-02-13T02:17:25.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6350: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2026-02-13T02:17:25.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6352: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 0 missing, 2 inconsistent objects' td/osd-scrub-repair/osd.1.log 2026-02-13T02:17:25.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6350: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2026-02-13T02:17:25.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6352: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 4 errors' td/osd-scrub-repair/osd.1.log 2026-02-13T02:17:25.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6359: TEST_corrupt_snapset_scrub_rep: '[' 0 '!=' 0 ']' 2026-02-13T02:17:25.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6365: TEST_corrupt_snapset_scrub_rep: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2026-02-13T02:17:26.068 INFO:tasks.workunit.client.0.trial112.stderr:pool 'csr_pool' removed 2026-02-13T02:17:26.079 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-13T02:17:26.079 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:17:26.079 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:17:26.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:17:26.080 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:17:26.080 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:17:26.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:17:26.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:17:26.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:17:26.393 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:17:26.393 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:17:26.395 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:17:26.396 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:17:26.397 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:17:26.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:17:26.398 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:17:26.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:17:26.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:17:26.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:17:26.400 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:17:26.401 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:17:26.402 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:17:26.403 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:17:26.414 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:17:26.414 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:17:26.414 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:17:26.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:17:26.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:17:26.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:17:26.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-13T02:17:26.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-13T02:17:26.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-13T02:17:26.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-13T02:17:26.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:17:26.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:17:26.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:17:26.418 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:17:26.418 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:17:26.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:17:26.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:17:26.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:17:26.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:17:26.421 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:17:26.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:17:26.422 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:17:26.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:17:26.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:17:26.424 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:17:26.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:17:26.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:17:26.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:17:26.426 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:17:26.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:17:26.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:17:26.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:17:26.430 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:17:26.431 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:17:26.431 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:17:26.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:17:26.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:17:26.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:17:26.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-13T02:17:26.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T02:17:26.434 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:17:26.434 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:17:26.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.35207 2026-02-13T02:17:26.436 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T02:17:26.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-13T02:17:26.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T02:17:26.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-13T02:17:26.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_dual_store_replicated_cluster td/osd-scrub-repair 2026-02-13T02:17:26.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6432: TEST_dual_store_replicated_cluster: local dir=td/osd-scrub-repair 2026-02-13T02:17:26.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6433: TEST_dual_store_replicated_cluster: local poolname=csr_pool 2026-02-13T02:17:26.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6434: TEST_dual_store_replicated_cluster: local total_objs=19 2026-02-13T02:17:26.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6435: TEST_dual_store_replicated_cluster: local extr_dbg=1 2026-02-13T02:17:26.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6437: TEST_dual_store_replicated_cluster: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2026-02-13T02:17:26.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-13T02:17:26.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-13T02:17:26.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-13T02:17:26.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-13T02:17:26.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-13T02:17:26.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2026-02-13T02:17:26.460 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-13T02:17:26.460 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:17:26.460 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:17:26.460 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:17:26.460 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:17:26.460 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:17:26.461 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:17:26.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2026-02-13T02:17:26.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-13T02:17:26.491 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-13T02:17:26.491 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:17:26.491 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:17:26.491 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-13T02:17:26.492 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-13T02:17:26.492 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:17:26.492 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:17:26.492 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:17:26.492 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:17:26.492 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:17:26.492 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:17:26.493 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:17:26.493 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:17:26.493 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get fsid 2026-02-13T02:17:26.549 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-13T02:17:26.549 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:17:26.549 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:17:26.549 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-13T02:17:26.550 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-13T02:17:26.550 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:17:26.551 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:17:26.551 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:17:26.551 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:17:26.551 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:17:26.551 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:17:26.551 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:17:26.552 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:17:26.552 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get mon_host 2026-02-13T02:17:26.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6438: TEST_dual_store_replicated_cluster: run_mgr td/osd-scrub-repair x --mgr_stats_period=1 2026-02-13T02:17:26.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-13T02:17:26.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-13T02:17:26.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-13T02:17:26.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-13T02:17:26.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-13T02:17:26.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-13T02:17:26.704 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-13T02:17:26.705 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:17:26.705 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:17:26.705 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:17:26.705 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:17:26.706 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:17:26.706 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:17:26.706 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:17:26.707 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mgr_stats_period=1 2026-02-13T02:17:26.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6439: TEST_dual_store_replicated_cluster: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 ' 2026-02-13T02:17:26.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6440: TEST_dual_store_replicated_cluster: ceph_osd_args+='--osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 ' 2026-02-13T02:17:26.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6441: TEST_dual_store_replicated_cluster: ceph_osd_args+='--osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 ' 2026-02-13T02:17:26.727 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6442: TEST_dual_store_replicated_cluster: seq 0 1 2026-02-13T02:17:26.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6442: TEST_dual_store_replicated_cluster: for osd in $(seq 0 1) 2026-02-13T02:17:26.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6444: TEST_dual_store_replicated_cluster: run_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:17:26.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:17:26.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:17:26.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-13T02:17:26.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:17:26.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:17:26.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:17:26.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:17:26.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:17:26.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:17:26.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:17:26.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:17:26.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:17:26.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:17:26.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:17:26.732 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:17:26.732 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:17:26.732 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:17:26.732 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:17:26.732 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:17:26.732 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:17:26.732 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:17:26.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:17:26.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:17:26.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:17:26.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:17:26.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:17:26.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:17:26.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:17:26.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:17:26.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:17:26.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:17:26.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:17:26.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-13T02:17:26.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:17:26.734 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:17:26.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4f8d5ce6-a345-424d-8fc8-9eb1e4df45a1 2026-02-13T02:17:26.734 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 4f8d5ce6-a345-424d-8fc8-9eb1e4df45a1 2026-02-13T02:17:26.735 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 4f8d5ce6-a345-424d-8fc8-9eb1e4df45a1' 2026-02-13T02:17:26.735 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:17:26.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC2iY5piFjKKxAA2swxFT1fkPoZDVossjwbMw== 2026-02-13T02:17:26.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC2iY5piFjKKxAA2swxFT1fkPoZDVossjwbMw=="}' 2026-02-13T02:17:26.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4f8d5ce6-a345-424d-8fc8-9eb1e4df45a1 -i td/osd-scrub-repair/0/new.json 2026-02-13T02:17:26.835 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:17:26.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-13T02:17:26.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 --mkfs --key AQC2iY5piFjKKxAA2swxFT1fkPoZDVossjwbMw== --osd-uuid 4f8d5ce6-a345-424d-8fc8-9eb1e4df45a1 2026-02-13T02:17:26.861 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:17:26.849+0000 7f1b98eefa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:17:26.863 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:17:26.849+0000 7f1b98eefa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:17:26.864 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:17:26.849+0000 7f1b98eefa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:17:26.864 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:17:26.849+0000 7f1b98eefa40 -1 bdev(0x5595eb8aec00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-13T02:17:26.865 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:17:26.853+0000 7f1b98eefa40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-13T02:17:27.556 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-13T02:17:27.557 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:17:27.557 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T02:17:27.558 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-13T02:17:27.558 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:17:27.652 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T02:17:27.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-13T02:17:27.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:17:27.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:17:27.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:17:27.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:17:27.669 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:17:27.657+0000 7fe35a270a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:17:27.670 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:17:27.657+0000 7fe35a270a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:17:27.671 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:17:27.657+0000 7fe35a270a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:17:27.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-13T02:17:27.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:17:27.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:17:27.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:17:27.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:17:27.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:17:27.751 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:17:27.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:17:27.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:17:27.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:17:27.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:17:28.146 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:17:28.133+0000 7fe35a270a40 -1 Falling back to public interface 2026-02-13T02:17:28.770 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:17:28.757+0000 7fe35a270a40 -1 osd.0 0 log_to_monitors true 2026-02-13T02:17:28.845 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:17:28.845 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:17:28.845 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:17:28.845 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:17:28.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:17:28.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:17:28.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:17:29.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:17:29.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:17:29.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:17:29.969 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:17:29.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:17:29.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:17:30.129 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/461717053,v1:127.0.0.1:6801/461717053] [v2:127.0.0.1:6802/461717053,v1:127.0.0.1:6803/461717053] exists,up 4f8d5ce6-a345-424d-8fc8-9eb1e4df45a1 2026-02-13T02:17:30.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:17:30.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:17:30.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:17:30.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6442: TEST_dual_store_replicated_cluster: for osd in $(seq 0 1) 2026-02-13T02:17:30.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6444: TEST_dual_store_replicated_cluster: run_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:17:30.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:17:30.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:17:30.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-13T02:17:30.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:17:30.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:17:30.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:17:30.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:17:30.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:17:30.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:17:30.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:17:30.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:17:30.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:17:30.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:17:30.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:17:30.135 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:17:30.135 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:17:30.135 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:17:30.135 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:17:30.135 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:17:30.135 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:17:30.135 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:17:30.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:17:30.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:17:30.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:17:30.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:17:30.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:17:30.136 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:17:30.136 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:17:30.136 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:17:30.136 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:17:30.136 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:17:30.136 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:17:30.136 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-13T02:17:30.136 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:17:30.136 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:17:30.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=65f77df4-d31d-4013-880d-185b2abb9aaf 2026-02-13T02:17:30.137 INFO:tasks.workunit.client.0.trial112.stdout:add osd1 65f77df4-d31d-4013-880d-185b2abb9aaf 2026-02-13T02:17:30.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 65f77df4-d31d-4013-880d-185b2abb9aaf' 2026-02-13T02:17:30.138 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:17:30.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC6iY5pUqkfCBAALPC43jV2i7x1hKS5xzsxeg== 2026-02-13T02:17:30.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC6iY5pUqkfCBAALPC43jV2i7x1hKS5xzsxeg=="}' 2026-02-13T02:17:30.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 65f77df4-d31d-4013-880d-185b2abb9aaf -i td/osd-scrub-repair/1/new.json 2026-02-13T02:17:30.311 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:17:30.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-13T02:17:30.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 --mkfs --key AQC6iY5pUqkfCBAALPC43jV2i7x1hKS5xzsxeg== --osd-uuid 65f77df4-d31d-4013-880d-185b2abb9aaf 2026-02-13T02:17:30.340 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:17:30.325+0000 7fe13a472a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:17:30.341 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:17:30.329+0000 7fe13a472a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:17:30.342 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:17:30.329+0000 7fe13a472a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:17:30.343 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:17:30.329+0000 7fe13a472a40 -1 bdev(0x55e816a08c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-13T02:17:30.343 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:17:30.329+0000 7fe13a472a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-13T02:17:30.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-13T02:17:30.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:17:30.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-13T02:17:30.953 INFO:tasks.workunit.client.0.trial112.stdout:adding osd1 key to auth repository 2026-02-13T02:17:30.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:17:31.157 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T02:17:31.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-13T02:17:31.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:17:31.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:17:31.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:17:31.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:17:31.184 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:17:31.169+0000 7f30dc630a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:17:31.186 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:17:31.173+0000 7f30dc630a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:17:31.187 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:17:31.173+0000 7f30dc630a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:17:31.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-13T02:17:31.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:17:31.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:17:31.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:17:31.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:17:31.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:17:31.323 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:17:31.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:17:31.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:17:31.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:17:31.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:17:31.530 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:17:31.517+0000 7f30dc630a40 -1 Falling back to public interface 2026-02-13T02:17:32.126 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:17:32.113+0000 7f30dc630a40 -1 osd.1 0 log_to_monitors true 2026-02-13T02:17:32.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:17:32.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:17:32.482 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:17:32.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:17:32.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:17:32.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:17:32.645 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2096470943,v1:127.0.0.1:6811/2096470943] [v2:127.0.0.1:6812/2096470943,v1:127.0.0.1:6813/2096470943] exists,up 65f77df4-d31d-4013-880d-185b2abb9aaf 2026-02-13T02:17:32.645 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:17:32.646 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:17:32.646 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:17:32.646 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6447: TEST_dual_store_replicated_cluster: create_rbd_pool 2026-02-13T02:17:32.646 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-13T02:17:32.797 INFO:tasks.workunit.client.0.trial112.stderr:pool 'rbd' does not exist 2026-02-13T02:17:32.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-13T02:17:32.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-13T02:17:33.029 INFO:tasks.workunit.client.0.trial112.stderr:pool 'rbd' created 2026-02-13T02:17:33.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:17:34.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-13T02:17:34.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6448: TEST_dual_store_replicated_cluster: wait_for_clean 2026-02-13T02:17:34.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:17:34.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:17:34.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:17:34.322 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:17:34.322 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:17:34.322 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:17:34.322 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:17:34.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:17:34.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:17:34.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:17:34.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:17:34.439 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:17:34.439 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:17:34.439 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:17:34.439 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:17:34.601 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:17:34.601 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:17:34.601 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:17:34.601 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:17:34.601 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:17:34.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836487 2026-02-13T02:17:34.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836487 2026-02-13T02:17:34.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487' 2026-02-13T02:17:34.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:17:34.676 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:17:34.747 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672963 2026-02-13T02:17:34.747 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672963 2026-02-13T02:17:34.747 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487 1-42949672963' 2026-02-13T02:17:34.747 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:17:34.748 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836487 2026-02-13T02:17:34.748 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:17:34.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:17:34.750 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836487 2026-02-13T02:17:34.750 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:17:34.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836487 2026-02-13T02:17:34.752 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836487 2026-02-13T02:17:34.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2026-02-13T02:17:34.752 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:17:34.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836487 2026-02-13T02:17:34.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:17:35.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:17:35.922 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:17:36.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836487 2026-02-13T02:17:36.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:17:36.083 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672963 2026-02-13T02:17:36.084 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:17:36.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:17:36.086 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672963 2026-02-13T02:17:36.086 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:17:36.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672963 2026-02-13T02:17:36.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2026-02-13T02:17:36.088 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 42949672963 2026-02-13T02:17:36.088 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:17:36.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672963 -lt 42949672963 2026-02-13T02:17:36.262 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:17:36.262 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:17:36.263 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:17:36.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-13T02:17:36.455 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:17:36.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:17:36.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:17:36.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:17:36.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:17:36.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:17:36.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:17:36.610 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-13T02:17:36.610 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:17:36.611 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:17:36.611 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:17:36.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-13T02:17:36.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:17:36.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:17:36.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6450: TEST_dual_store_replicated_cluster: create_pool foo 1 2026-02-13T02:17:36.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 1 2026-02-13T02:17:36.981 INFO:tasks.workunit.client.0.trial112.stderr:pool 'foo' created 2026-02-13T02:17:36.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:17:37.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6451: TEST_dual_store_replicated_cluster: create_pool csr_pool 1 1 2026-02-13T02:17:37.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create csr_pool 1 1 2026-02-13T02:17:38.206 INFO:tasks.workunit.client.0.trial112.stderr:pool 'csr_pool' created 2026-02-13T02:17:38.218 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:17:39.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6452: TEST_dual_store_replicated_cluster: wait_for_clean 2026-02-13T02:17:39.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:17:39.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:17:39.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:17:39.221 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:17:39.221 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:17:39.222 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:17:39.222 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:17:39.222 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:17:39.222 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:17:39.343 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:17:39.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:17:39.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:17:39.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:17:39.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:17:39.344 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:17:39.508 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:17:39.509 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:17:39.509 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:17:39.509 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:17:39.509 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:17:39.585 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836493 2026-02-13T02:17:39.585 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836493 2026-02-13T02:17:39.585 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493' 2026-02-13T02:17:39.585 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:17:39.586 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:17:39.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672969 2026-02-13T02:17:39.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672969 2026-02-13T02:17:39.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-42949672969' 2026-02-13T02:17:39.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:17:39.661 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836493 2026-02-13T02:17:39.661 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:17:39.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:17:39.662 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836493 2026-02-13T02:17:39.662 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:17:39.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836493 2026-02-13T02:17:39.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2026-02-13T02:17:39.665 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836493 2026-02-13T02:17:39.665 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:17:39.827 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836493 2026-02-13T02:17:39.827 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:17:40.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:17:40.829 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:17:40.988 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836493 2026-02-13T02:17:40.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:17:41.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:17:41.990 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:17:42.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836494 -lt 21474836493 2026-02-13T02:17:42.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:17:42.153 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672969 2026-02-13T02:17:42.154 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:17:42.155 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:17:42.156 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672969 2026-02-13T02:17:42.156 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:17:42.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672969 2026-02-13T02:17:42.158 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 42949672969 2026-02-13T02:17:42.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2026-02-13T02:17:42.159 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:17:42.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672970 -lt 42949672969 2026-02-13T02:17:42.329 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:17:42.329 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:17:42.329 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:17:42.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:17:42.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:17:42.526 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:17:42.527 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:17:42.527 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:17:42.527 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:17:42.527 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:17:42.528 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:17:42.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:17:42.691 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:17:42.691 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:17:42.691 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:17:42.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:17:42.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:17:42.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:17:42.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6454: TEST_dual_store_replicated_cluster: ceph osd pool set csr_pool noscrub 1 2026-02-13T02:17:43.093 INFO:tasks.workunit.client.0.trial112.stderr:set pool 3 noscrub to 1 2026-02-13T02:17:43.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6455: TEST_dual_store_replicated_cluster: ceph osd pool set csr_pool nodeep-scrub 1 2026-02-13T02:17:43.297 INFO:tasks.workunit.client.0.trial112.stderr:set pool 3 nodeep-scrub to 1 2026-02-13T02:17:43.308 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: seq 1 19 2026-02-13T02:17:43.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-13T02:17:43.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ1 2026-02-13T02:17:43.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ1 2026-02-13T02:17:43.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:17:43.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-13T02:17:43.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ1 2026-02-13T02:17:43.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:17:43.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:17:43.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:17:43.500 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:17:43.511 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:17:43.700 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:17:43.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:17:43.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:17:43.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:17:43.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2026-02-13T02:17:43.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2026-02-13T02:17:43.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-13T02:17:43.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ2 2026-02-13T02:17:43.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ2 2026-02-13T02:17:43.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:17:43.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-13T02:17:43.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ2 2026-02-13T02:17:43.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:17:43.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:17:43.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:17:43.990 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:17:44.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:17:44.195 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:17:44.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:17:44.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:17:44.206 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:17:44.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2026-02-13T02:17:44.251 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2026-02-13T02:17:44.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-13T02:17:44.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ3 2026-02-13T02:17:44.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ3 2026-02-13T02:17:44.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:17:44.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-13T02:17:44.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ3 2026-02-13T02:17:44.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:17:44.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:17:44.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:17:44.486 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:17:44.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:17:44.691 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:17:44.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:17:44.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:17:44.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ3 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:17:44.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ3 hdr-ROBJ3 2026-02-13T02:17:44.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ3 key-ROBJ3 val-ROBJ3 2026-02-13T02:17:44.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-13T02:17:44.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ4 2026-02-13T02:17:44.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ4 2026-02-13T02:17:44.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:17:44.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-13T02:17:44.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ4 2026-02-13T02:17:44.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:17:44.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:17:44.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:17:44.989 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:17:45.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:17:45.196 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:17:45.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:17:45.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:17:45.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ4 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:17:45.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ4 hdr-ROBJ4 2026-02-13T02:17:45.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ4 key-ROBJ4 val-ROBJ4 2026-02-13T02:17:45.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-13T02:17:45.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ5 2026-02-13T02:17:45.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ5 2026-02-13T02:17:45.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:17:45.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-13T02:17:45.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ5 2026-02-13T02:17:45.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:17:45.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:17:45.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:17:45.495 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:17:45.508 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:17:45.701 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:17:45.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:17:45.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:17:45.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ5 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:17:45.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ5 hdr-ROBJ5 2026-02-13T02:17:45.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ5 key-ROBJ5 val-ROBJ5 2026-02-13T02:17:45.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-13T02:17:45.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ6 2026-02-13T02:17:45.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ6 2026-02-13T02:17:45.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:17:45.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-13T02:17:45.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ6 2026-02-13T02:17:45.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:17:45.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:17:45.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:17:46.004 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:17:46.015 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:17:46.210 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:17:46.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:17:46.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:17:46.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ6 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:17:46.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ6 hdr-ROBJ6 2026-02-13T02:17:46.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ6 key-ROBJ6 val-ROBJ6 2026-02-13T02:17:46.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-13T02:17:46.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ7 2026-02-13T02:17:46.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ7 2026-02-13T02:17:46.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:17:46.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-13T02:17:46.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ7 2026-02-13T02:17:46.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:17:46.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:17:46.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:17:46.507 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:17:46.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:17:46.712 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:17:46.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:17:46.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:17:46.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ7 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:17:46.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ7 hdr-ROBJ7 2026-02-13T02:17:46.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ7 key-ROBJ7 val-ROBJ7 2026-02-13T02:17:46.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-13T02:17:46.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ8 2026-02-13T02:17:46.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ8 2026-02-13T02:17:46.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:17:46.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-13T02:17:46.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ8 2026-02-13T02:17:46.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:17:46.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:17:46.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:17:47.017 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:17:47.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:17:47.222 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:17:47.233 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:17:47.233 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:17:47.233 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ8 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:17:47.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ8 hdr-ROBJ8 2026-02-13T02:17:47.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ8 key-ROBJ8 val-ROBJ8 2026-02-13T02:17:47.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-13T02:17:47.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ9 2026-02-13T02:17:47.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ9 2026-02-13T02:17:47.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:17:47.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-13T02:17:47.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ9 2026-02-13T02:17:47.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:17:47.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:17:47.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:17:47.522 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:17:47.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:17:47.728 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:17:47.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:17:47.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:17:47.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:17:47.768 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ9 hdr-ROBJ9 2026-02-13T02:17:47.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ9 key-ROBJ9 val-ROBJ9 2026-02-13T02:17:47.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-13T02:17:47.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ10 2026-02-13T02:17:47.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ10 2026-02-13T02:17:47.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:17:47.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-13T02:17:47.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ10 2026-02-13T02:17:47.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:17:47.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:17:47.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:17:48.024 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:17:48.037 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:17:48.230 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:17:48.242 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:17:48.242 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:17:48.242 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ10 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:17:48.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ10 hdr-ROBJ10 2026-02-13T02:17:48.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ10 key-ROBJ10 val-ROBJ10 2026-02-13T02:17:48.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-13T02:17:48.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ11 2026-02-13T02:17:48.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ11 2026-02-13T02:17:48.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:17:48.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-13T02:17:48.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ11 2026-02-13T02:17:48.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:17:48.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:17:48.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:17:48.517 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:17:48.527 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:17:48.722 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:17:48.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:17:48.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:17:48.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ11 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:17:48.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ11 hdr-ROBJ11 2026-02-13T02:17:48.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ11 key-ROBJ11 val-ROBJ11 2026-02-13T02:17:48.801 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-13T02:17:48.801 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ12 2026-02-13T02:17:48.801 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ12 2026-02-13T02:17:48.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:17:48.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-13T02:17:48.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ12 2026-02-13T02:17:48.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:17:48.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:17:48.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:17:49.006 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:17:49.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:17:49.211 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:17:49.222 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:17:49.223 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:17:49.223 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ12 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:17:49.243 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ12 hdr-ROBJ12 2026-02-13T02:17:49.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ12 key-ROBJ12 val-ROBJ12 2026-02-13T02:17:49.295 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-13T02:17:49.295 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ13 2026-02-13T02:17:49.295 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ13 2026-02-13T02:17:49.295 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:17:49.295 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-13T02:17:49.295 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ13 2026-02-13T02:17:49.295 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:17:49.295 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:17:49.295 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:17:49.505 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:17:49.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:17:49.710 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:17:49.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:17:49.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:17:49.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ13 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:17:49.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ13 hdr-ROBJ13 2026-02-13T02:17:49.773 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ13 key-ROBJ13 val-ROBJ13 2026-02-13T02:17:49.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-13T02:17:49.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ14 2026-02-13T02:17:49.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ14 2026-02-13T02:17:49.801 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:17:49.801 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-13T02:17:49.801 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ14 2026-02-13T02:17:49.801 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:17:49.801 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:17:49.801 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:17:50.012 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:17:50.024 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:17:50.218 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:17:50.229 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:17:50.229 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:17:50.229 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ14 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:17:50.255 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ14 hdr-ROBJ14 2026-02-13T02:17:50.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ14 key-ROBJ14 val-ROBJ14 2026-02-13T02:17:50.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-13T02:17:50.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ15 2026-02-13T02:17:50.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ15 2026-02-13T02:17:50.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:17:50.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-13T02:17:50.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ15 2026-02-13T02:17:50.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:17:50.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:17:50.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:17:50.513 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:17:50.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:17:50.718 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:17:50.729 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:17:50.729 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:17:50.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ15 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:17:50.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ15 hdr-ROBJ15 2026-02-13T02:17:50.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ15 key-ROBJ15 val-ROBJ15 2026-02-13T02:17:50.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-13T02:17:50.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ16 2026-02-13T02:17:50.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ16 2026-02-13T02:17:50.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:17:50.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-13T02:17:50.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ16 2026-02-13T02:17:50.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:17:50.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:17:50.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:17:51.017 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:17:51.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:17:51.222 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:17:51.233 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:17:51.233 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:17:51.233 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ16 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:17:51.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ16 hdr-ROBJ16 2026-02-13T02:17:51.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ16 key-ROBJ16 val-ROBJ16 2026-02-13T02:17:51.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-13T02:17:51.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ17 2026-02-13T02:17:51.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ17 2026-02-13T02:17:51.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:17:51.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-13T02:17:51.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ17 2026-02-13T02:17:51.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:17:51.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:17:51.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:17:51.524 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:17:51.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:17:51.730 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:17:51.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:17:51.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:17:51.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ17 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:17:51.768 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ17 hdr-ROBJ17 2026-02-13T02:17:51.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ17 key-ROBJ17 val-ROBJ17 2026-02-13T02:17:51.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-13T02:17:51.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ18 2026-02-13T02:17:51.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ18 2026-02-13T02:17:51.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:17:51.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-13T02:17:51.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ18 2026-02-13T02:17:51.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:17:51.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:17:51.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:17:52.020 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:17:52.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:17:52.225 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:17:52.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:17:52.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:17:52.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ18 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:17:52.258 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ18 hdr-ROBJ18 2026-02-13T02:17:52.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ18 key-ROBJ18 val-ROBJ18 2026-02-13T02:17:52.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-13T02:17:52.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ19 2026-02-13T02:17:52.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ19 2026-02-13T02:17:52.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:17:52.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-13T02:17:52.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ19 2026-02-13T02:17:52.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:17:52.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:17:52.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:17:52.512 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:17:52.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:17:52.717 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:17:52.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:17:52.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:17:52.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:17:52.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ19 hdr-ROBJ19 2026-02-13T02:17:52.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ19 key-ROBJ19 val-ROBJ19 2026-02-13T02:17:52.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6466: TEST_dual_store_replicated_cluster: dd if=/dev/zero of=td/osd-scrub-repair/new.ROBJ19 bs=1024 count=1025 2026-02-13T02:17:52.812 INFO:tasks.workunit.client.0.trial112.stderr:1025+0 records in 2026-02-13T02:17:52.813 INFO:tasks.workunit.client.0.trial112.stderr:1025+0 records out 2026-02-13T02:17:52.813 INFO:tasks.workunit.client.0.trial112.stderr:1049600 bytes (1.0 MB, 1.0 MiB) copied, 0.00400833 s, 262 MB/s 2026-02-13T02:17:52.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6467: TEST_dual_store_replicated_cluster: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/new.ROBJ19 2026-02-13T02:17:52.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6468: TEST_dual_store_replicated_cluster: rm -f td/osd-scrub-repair/new.ROBJ19 2026-02-13T02:17:52.843 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6470: TEST_dual_store_replicated_cluster: get_pg csr_pool ROBJ0 2026-02-13T02:17:52.843 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=csr_pool 2026-02-13T02:17:52.843 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=ROBJ0 2026-02-13T02:17:52.844 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map csr_pool ROBJ0 2026-02-13T02:17:52.844 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T02:17:53.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6470: TEST_dual_store_replicated_cluster: local pg=3.0 2026-02-13T02:17:53.003 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6471: TEST_dual_store_replicated_cluster: get_primary csr_pool ROBJ0 2026-02-13T02:17:53.003 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=csr_pool 2026-02-13T02:17:53.003 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=ROBJ0 2026-02-13T02:17:53.004 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map csr_pool ROBJ0 2026-02-13T02:17:53.004 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T02:17:53.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6471: TEST_dual_store_replicated_cluster: local primary=1 2026-02-13T02:17:53.165 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6474: TEST_dual_store_replicated_cluster: get_asok_path osd.0 2026-02-13T02:17:53.166 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-13T02:17:53.166 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-13T02:17:53.167 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:17:53.167 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:17:53.167 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:17:53.167 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-osd.0.asok 2026-02-13T02:17:53.167 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6474: TEST_dual_store_replicated_cluster: CEPH_ARGS= 2026-02-13T02:17:53.167 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6474: TEST_dual_store_replicated_cluster: ceph daemon /tmp/ceph-asok.35207/ceph-osd.0.asok config set osd_deep_scrub_update_digest_min_age 0 2026-02-13T02:17:53.230 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:17:53.230 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2026-02-13T02:17:53.230 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:17:53.237 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6476: TEST_dual_store_replicated_cluster: get_asok_path osd.1 2026-02-13T02:17:53.237 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-13T02:17:53.237 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-13T02:17:53.238 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:17:53.238 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:17:53.238 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:17:53.238 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-osd.1.asok 2026-02-13T02:17:53.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6476: TEST_dual_store_replicated_cluster: CEPH_ARGS= 2026-02-13T02:17:53.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6476: TEST_dual_store_replicated_cluster: ceph daemon /tmp/ceph-asok.35207/ceph-osd.1.asok config set osd_deep_scrub_update_digest_min_age 0 2026-02-13T02:17:53.292 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:17:53.292 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2026-02-13T02:17:53.292 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:17:53.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6478: TEST_dual_store_replicated_cluster: pg_deep_scrub 3.0 2026-02-13T02:17:53.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1932: pg_deep_scrub: local pgid=3.0 2026-02-13T02:17:53.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: pg_deep_scrub: wait_for_pg_clean 3.0 2026-02-13T02:17:53.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-13T02:17:53.300 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-13T02:17:53.300 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:17:53.300 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:17:53.300 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:17:53.300 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:17:53.300 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:17:53.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-13T02:17:53.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-13T02:17:53.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-13T02:17:53.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-13T02:17:53.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:17:53.591 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:17:53.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:17:53.755 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:17:53.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:17:53.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:17:53.755 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:17:53.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836508 2026-02-13T02:17:53.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836508 2026-02-13T02:17:53.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836508' 2026-02-13T02:17:53.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:17:53.833 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:17:53.904 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672984 2026-02-13T02:17:53.904 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672984 2026-02-13T02:17:53.904 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836508 1-42949672984' 2026-02-13T02:17:53.904 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:17:53.905 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836508 2026-02-13T02:17:53.905 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:17:53.906 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:17:53.906 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836508 2026-02-13T02:17:53.907 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:17:53.908 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836508 2026-02-13T02:17:53.908 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2026-02-13T02:17:53.909 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836508 2026-02-13T02:17:53.909 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:17:54.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836506 -lt 21474836508 2026-02-13T02:17:54.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:17:55.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:17:55.071 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:17:55.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836509 -lt 21474836508 2026-02-13T02:17:55.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:17:55.237 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672984 2026-02-13T02:17:55.237 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:17:55.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:17:55.240 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672984 2026-02-13T02:17:55.240 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:17:55.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672984 2026-02-13T02:17:55.242 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2026-02-13T02:17:55.242 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 42949672984 2026-02-13T02:17:55.242 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:17:55.402 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672985 -lt 42949672984 2026-02-13T02:17:55.402 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-13T02:17:55.402 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-13T02:17:55.402 INFO:tasks.workunit.client.0.trial112.stdout:#---------- 3.0 loop 0 2026-02-13T02:17:55.403 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-13T02:17:55.403 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-13T02:17:55.403 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-13T02:17:55.403 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-13T02:17:55.404 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-13T02:17:55.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-13T02:17:55.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-13T02:17:55.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-13T02:17:55.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-13T02:17:55.480 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-13T02:17:55.480 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:17:55.480 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:17:55.480 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:17:55.481 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-13T02:17:55.643 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: local last_scrub=2026-02-13T02:17:38.195712+0000 2026-02-13T02:17:55.643 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: pg_deep_scrub: ceph pg deep-scrub 3.0 2026-02-13T02:17:55.793 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2026-02-13T02:17:55.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: pg_deep_scrub: wait_for_scrub 3.0 2026-02-13T02:17:38.195712+0000 last_deep_scrub_stamp 2026-02-13T02:17:55.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-13T02:17:55.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:17:38.195712+0000 2026-02-13T02:17:55.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_deep_scrub_stamp 2026-02-13T02:17:55.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:17:55.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:17:55.806 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-13T02:17:55.806 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:17:55.806 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:17:55.806 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:17:55.806 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-13T02:17:55.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:17:38.195712+0000 '>' 2026-02-13T02:17:38.195712+0000 2026-02-13T02:17:55.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:17:56.970 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:17:56.971 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:17:56.971 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-13T02:17:56.971 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:17:56.972 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:17:56.972 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-13T02:17:56.972 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:17:57.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:17:56.280931+0000 '>' 2026-02-13T02:17:38.195712+0000 2026-02-13T02:17:57.136 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:17:57.136 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: seq 1 19 2026-02-13T02:17:57.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-13T02:17:57.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ1 2026-02-13T02:17:57.137 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 1 % 2 2026-02-13T02:17:57.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=1 2026-02-13T02:17:57.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-13T02:17:57.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6489: TEST_dual_store_replicated_cluster: local payload=UVWXYZZZ 2026-02-13T02:17:57.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6490: TEST_dual_store_replicated_cluster: echo UVWXYZZZ 2026-02-13T02:17:57.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6491: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-13T02:17:57.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:17:57.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:17:57.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:17:57.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:17:57.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-13T02:17:57.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:17:57.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:17:57.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:17:57.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:17:57.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:17:57.141 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:17:57.141 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:17:57.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:17:57.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:17:57.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:17:57.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:17:57.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-13T02:17:57.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:17:57.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:17:57.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:17:57.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:17:57.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:17:57.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-13T02:17:57.611 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 3.0_head,#-5:00000000:::deep_scrub_3.0:head#, (61) No data available 2026-02-13T02:17:57.611 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 3.0_head,#-5:00000000:::scrub_3.0:head#, (61) No data available 2026-02-13T02:17:57.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:17:57.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:17:57.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:17:57.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:17:57.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:17:57.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:17:57.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:17:57.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:17:57.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:17:57.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:17:57.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:17:57.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:17:57.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:17:57.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:17:57.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:17:57.687 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:17:57.687 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:17:57.687 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:17:57.687 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:17:57.688 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:17:57.688 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:17:57.689 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:17:57.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:17:57.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:17:57.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:17:57.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:17:57.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:17:57.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:17:57.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:17:57.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:17:57.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:17:57.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-13T02:17:57.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:17:57.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:17:57.691 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:17:57.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:17:57.691 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:17:57.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:17:57.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:17:57.692 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:17:57.692 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:17:57.709 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:17:57.697+0000 7f6cd3fd1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:17:57.710 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:17:57.697+0000 7f6cd3fd1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:17:57.711 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:17:57.697+0000 7f6cd3fd1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:17:57.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:17:57.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:17:57.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:17:57.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:17:57.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:17:57.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:17:57.854 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:17:57.854 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:17:57.854 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:17:57.854 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:17:58.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:17:58.150 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:17:58.137+0000 7f6cd3fd1a40 -1 Falling back to public interface 2026-02-13T02:17:58.796 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:17:58.781+0000 7f6cd3fd1a40 -1 osd.1 63 log_to_monitors true 2026-02-13T02:17:59.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:17:59.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:17:59.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:17:59.013 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:17:59.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:17:59.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:17:59.174 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 67 up_thru 67 down_at 64 last_clean_interval [10,63) [v2:127.0.0.1:6810/2596164432,v1:127.0.0.1:6811/2596164432] [v2:127.0.0.1:6812/2596164432,v1:127.0.0.1:6813/2596164432] exists,up 65f77df4-d31d-4013-880d-185b2abb9aaf 2026-02-13T02:17:59.175 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:17:59.175 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:17:59.175 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:17:59.175 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:17:59.175 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:17:59.175 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:17:59.175 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:17:59.175 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:17:59.176 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:17:59.176 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:17:59.177 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:17:59.177 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:17:59.177 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:17:59.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:17:59.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:17:59.295 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:17:59.295 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:17:59.295 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:17:59.295 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:17:59.456 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:17:59.456 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:17:59.456 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:17:59.456 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:17:59.457 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:17:59.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836515 2026-02-13T02:17:59.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836515 2026-02-13T02:17:59.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836515' 2026-02-13T02:17:59.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:17:59.530 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:17:59.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=287762808834 2026-02-13T02:17:59.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 287762808834 2026-02-13T02:17:59.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836515 1-287762808834' 2026-02-13T02:17:59.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:17:59.605 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836515 2026-02-13T02:17:59.605 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:17:59.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:17:59.607 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836515 2026-02-13T02:17:59.607 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:17:59.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836515 2026-02-13T02:17:59.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836515' 2026-02-13T02:17:59.609 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 21474836515 2026-02-13T02:17:59.609 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:17:59.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836513 -lt 21474836515 2026-02-13T02:17:59.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:18:00.773 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:18:00.774 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:18:00.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836513 -lt 21474836515 2026-02-13T02:18:00.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:18:01.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:18:01.937 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:18:02.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836516 -lt 21474836515 2026-02-13T02:18:02.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:18:02.104 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-287762808834 2026-02-13T02:18:02.104 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:18:02.105 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:18:02.106 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-287762808834 2026-02-13T02:18:02.107 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:18:02.108 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=287762808834 2026-02-13T02:18:02.108 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 287762808834' 2026-02-13T02:18:02.108 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 287762808834 2026-02-13T02:18:02.109 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:18:02.270 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 287762808836 -lt 287762808834 2026-02-13T02:18:02.270 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:18:02.271 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:18:02.271 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:18:02.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:18:02.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:18:02.470 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:18:02.470 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:18:02.470 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:18:02.470 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:18:02.470 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:18:02.470 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:18:02.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:18:02.630 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:18:02.631 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:18:02.631 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:18:02.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:18:02.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:18:02.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:18:02.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-13T02:18:02.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ2 2026-02-13T02:18:02.829 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 2 % 2 2026-02-13T02:18:02.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=0 2026-02-13T02:18:02.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-13T02:18:02.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6496: TEST_dual_store_replicated_cluster: local payload=UVWXYZ 2026-02-13T02:18:02.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6497: TEST_dual_store_replicated_cluster: echo UVWXYZ 2026-02-13T02:18:02.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6498: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-13T02:18:02.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:18:02.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:18:02.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T02:18:02.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:18:02.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-13T02:18:02.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:18:02.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:18:02.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T02:18:02.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:18:02.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T02:18:02.833 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:18:02.833 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:18:02.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:18:02.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:18:02.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:18:03.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:18:03.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-13T02:18:03.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:18:03.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:18:03.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:18:03.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:18:03.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:18:03.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-13T02:18:03.416 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:18:03.416 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:18:03.416 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:18:03.416 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:18:03.416 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:18:03.416 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:18:03.416 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:18:03.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:18:03.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:18:03.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:18:03.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:18:03.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:18:03.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:18:03.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:18:03.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:18:03.417 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:18:03.417 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:18:03.417 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:18:03.418 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:18:03.418 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:18:03.418 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:18:03.418 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:18:03.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:18:03.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:18:03.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:18:03.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:18:03.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:18:03.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:18:03.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:18:03.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:18:03.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:18:03.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-13T02:18:03.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:18:03.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:18:03.420 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T02:18:03.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:18:03.420 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T02:18:03.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:18:03.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:18:03.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:18:03.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:18:03.440 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:03.429+0000 7f7a34659a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:18:03.442 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:03.429+0000 7f7a34659a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:18:03.443 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:03.429+0000 7f7a34659a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:18:03.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:18:03.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:18:03.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:18:03.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:18:03.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:18:03.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:18:03.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:18:03.579 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:18:03.580 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:18:03.580 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:18:03.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:18:03.794 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:03.781+0000 7f7a34659a40 -1 Falling back to public interface 2026-02-13T02:18:04.367 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:04.353+0000 7f7a34659a40 -1 osd.0 69 log_to_monitors true 2026-02-13T02:18:04.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:18:04.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:18:04.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:18:04.745 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:18:04.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:18:04.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:18:04.912 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 73 up_thru 73 down_at 70 last_clean_interval [5,69) [v2:127.0.0.1:6800/802830561,v1:127.0.0.1:6801/802830561] [v2:127.0.0.1:6802/802830561,v1:127.0.0.1:6803/802830561] exists,up 4f8d5ce6-a345-424d-8fc8-9eb1e4df45a1 2026-02-13T02:18:04.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:18:04.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:18:04.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:18:04.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:18:04.913 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:18:04.913 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:18:04.913 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:18:04.913 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:18:04.914 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:18:04.914 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:18:04.914 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:18:04.914 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:18:04.914 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:18:05.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:18:05.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:18:05.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:18:05.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:18:05.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:18:05.029 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:18:05.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:18:05.193 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:18:05.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:18:05.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:18:05.193 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:18:05.267 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=313532612610 2026-02-13T02:18:05.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 313532612610 2026-02-13T02:18:05.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-313532612610' 2026-02-13T02:18:05.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:18:05.268 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:18:05.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=287762808841 2026-02-13T02:18:05.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 287762808841 2026-02-13T02:18:05.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-313532612610 1-287762808841' 2026-02-13T02:18:05.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:18:05.343 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-313532612610 2026-02-13T02:18:05.343 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:18:05.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:18:05.345 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-313532612610 2026-02-13T02:18:05.345 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:18:05.347 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=313532612610 2026-02-13T02:18:05.347 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 313532612610' 2026-02-13T02:18:05.347 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 313532612610 2026-02-13T02:18:05.347 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:18:05.515 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 313532612610 2026-02-13T02:18:05.516 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:18:06.516 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:18:06.517 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:18:06.681 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 313532612610 2026-02-13T02:18:06.681 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:18:07.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:18:07.683 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:18:07.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 313532612612 -lt 313532612610 2026-02-13T02:18:07.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:18:07.844 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-287762808841 2026-02-13T02:18:07.844 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:18:07.845 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:18:07.846 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-287762808841 2026-02-13T02:18:07.847 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:18:07.848 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=287762808841 2026-02-13T02:18:07.848 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 287762808841' 2026-02-13T02:18:07.848 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 287762808841 2026-02-13T02:18:07.849 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:18:08.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 287762808843 -lt 287762808841 2026-02-13T02:18:08.013 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:18:08.013 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:18:08.014 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:18:08.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:18:08.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:18:08.210 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:18:08.210 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:18:08.210 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:18:08.210 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:18:08.210 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:18:08.210 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:18:08.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:18:08.372 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:18:08.372 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:18:08.372 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:18:08.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:18:08.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:18:08.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:18:08.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-13T02:18:08.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ3 2026-02-13T02:18:08.571 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 3 % 2 2026-02-13T02:18:08.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=1 2026-02-13T02:18:08.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-13T02:18:08.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6503: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ3 remove 2026-02-13T02:18:08.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:18:08.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:18:08.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:18:08.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:18:08.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ3 remove 2026-02-13T02:18:08.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:18:08.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:18:08.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:18:08.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:18:08.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:18:08.573 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:18:08.573 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:18:08.574 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:18:08.574 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:18:08.574 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:18:08.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:18:08.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ3 remove 2026-02-13T02:18:08.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:18:08.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:18:08.886 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:18:08.886 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:18:08.886 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:18:08.886 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ3 remove 2026-02-13T02:18:09.041 INFO:tasks.workunit.client.0.trial112.stdout:remove #3:f2a5b2a4:::ROBJ3:head# 2026-02-13T02:18:09.138 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:18:09.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:18:09.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:18:09.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:18:09.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:18:09.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:18:09.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:18:09.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:18:09.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:18:09.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:18:09.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:18:09.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:18:09.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:18:09.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:18:09.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:18:09.140 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:18:09.140 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:18:09.140 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:18:09.140 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:18:09.140 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:18:09.140 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:18:09.140 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:18:09.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:18:09.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:18:09.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:18:09.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:18:09.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:18:09.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:18:09.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:18:09.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:18:09.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:18:09.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-13T02:18:09.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:18:09.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:18:09.143 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:18:09.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:18:09.143 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:18:09.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:18:09.144 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:18:09.144 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:18:09.144 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:18:09.159 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:09.145+0000 7f67e9b3fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:18:09.160 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:09.145+0000 7f67e9b3fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:18:09.161 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:09.149+0000 7f67e9b3fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:18:09.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:18:09.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:18:09.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:18:09.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:18:09.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:18:09.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:18:09.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:18:09.305 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:18:09.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:18:09.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:18:09.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:18:09.606 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:09.593+0000 7f67e9b3fa40 -1 Falling back to public interface 2026-02-13T02:18:10.164 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:10.149+0000 7f67e9b3fa40 -1 osd.1 74 log_to_monitors true 2026-02-13T02:18:10.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:18:10.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:18:10.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:18:10.464 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:18:10.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:18:10.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:18:10.626 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 78 up_thru 78 down_at 75 last_clean_interval [67,74) [v2:127.0.0.1:6810/1860665572,v1:127.0.0.1:6811/1860665572] [v2:127.0.0.1:6812/1860665572,v1:127.0.0.1:6813/1860665572] exists,up 65f77df4-d31d-4013-880d-185b2abb9aaf 2026-02-13T02:18:10.627 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:18:10.627 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:18:10.627 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:18:10.627 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:18:10.627 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:18:10.627 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:18:10.627 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:18:10.628 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:18:10.628 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:18:10.628 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:18:10.629 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:18:10.629 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:18:10.629 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:18:10.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:18:10.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:18:10.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:18:10.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:18:10.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:18:10.752 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:18:10.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:18:10.911 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:18:10.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:18:10.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:18:10.912 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:18:10.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=313532612617 2026-02-13T02:18:10.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 313532612617 2026-02-13T02:18:10.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-313532612617' 2026-02-13T02:18:10.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:18:10.988 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:18:11.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=335007449090 2026-02-13T02:18:11.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 335007449090 2026-02-13T02:18:11.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-313532612617 1-335007449090' 2026-02-13T02:18:11.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:18:11.064 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-313532612617 2026-02-13T02:18:11.064 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:18:11.065 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:18:11.066 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-313532612617 2026-02-13T02:18:11.066 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:18:11.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=313532612617 2026-02-13T02:18:11.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 313532612617' 2026-02-13T02:18:11.068 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 313532612617 2026-02-13T02:18:11.069 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:18:11.232 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 313532612617 -lt 313532612617 2026-02-13T02:18:11.232 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:18:11.233 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-335007449090 2026-02-13T02:18:11.233 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:18:11.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:18:11.235 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-335007449090 2026-02-13T02:18:11.235 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:18:11.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=335007449090 2026-02-13T02:18:11.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 335007449090' 2026-02-13T02:18:11.237 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 335007449090 2026-02-13T02:18:11.237 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:18:11.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 335007449090 2026-02-13T02:18:11.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:18:12.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:18:12.400 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:18:12.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 335007449090 2026-02-13T02:18:12.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:18:13.568 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:18:13.568 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:18:13.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 335007449092 -lt 335007449090 2026-02-13T02:18:13.734 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:18:13.734 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:18:13.735 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:18:13.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:18:13.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:18:13.932 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:18:13.932 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:18:13.932 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:18:13.932 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:18:13.932 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:18:13.933 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:18:14.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:18:14.093 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:18:14.093 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:18:14.094 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:18:14.290 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:18:14.291 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:18:14.291 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:18:14.291 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-13T02:18:14.291 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ4 2026-02-13T02:18:14.291 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 4 % 2 2026-02-13T02:18:14.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=0 2026-02-13T02:18:14.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-13T02:18:14.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6508: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2026-02-13T02:18:14.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:18:14.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:18:14.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T02:18:14.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:18:14.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2026-02-13T02:18:14.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:18:14.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:18:14.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T02:18:14.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:18:14.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T02:18:14.295 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:18:14.295 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:18:14.295 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:18:14.295 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:18:14.295 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:18:14.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:18:14.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2026-02-13T02:18:14.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:18:14.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:18:14.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:18:14.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:18:14.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:18:14.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2026-02-13T02:18:14.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:18:14.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:18:14.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:18:14.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:18:14.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:18:14.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:18:14.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:18:14.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:18:14.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:18:14.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:18:14.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:18:14.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:18:14.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:18:14.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:18:14.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:18:14.881 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:18:14.881 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:18:14.881 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:18:14.881 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:18:14.881 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:18:14.881 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:18:14.881 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:18:14.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:18:14.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:18:14.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:18:14.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:18:14.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:18:14.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:18:14.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:18:14.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:18:14.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:18:14.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-13T02:18:14.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:18:14.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:18:14.884 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T02:18:14.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:18:14.884 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T02:18:14.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:18:14.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:18:14.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:18:14.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:18:14.899 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:14.885+0000 7f41ba422a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:18:14.900 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:14.889+0000 7f41ba422a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:18:14.901 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:14.889+0000 7f41ba422a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:18:15.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:18:15.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:18:15.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:18:15.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:18:15.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:18:15.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:18:15.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:18:15.044 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:18:15.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:18:15.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:18:15.206 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:18:15.250 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:15.237+0000 7f41ba422a40 -1 Falling back to public interface 2026-02-13T02:18:15.823 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:15.809+0000 7f41ba422a40 -1 osd.0 79 log_to_monitors true 2026-02-13T02:18:16.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:18:16.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:18:16.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:18:16.208 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:18:16.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:18:16.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:18:16.364 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 83 up_thru 83 down_at 80 last_clean_interval [73,79) [v2:127.0.0.1:6800/1043426850,v1:127.0.0.1:6801/1043426850] [v2:127.0.0.1:6802/1043426850,v1:127.0.0.1:6803/1043426850] exists,up 4f8d5ce6-a345-424d-8fc8-9eb1e4df45a1 2026-02-13T02:18:16.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:18:16.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:18:16.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:18:16.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:18:16.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:18:16.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:18:16.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:18:16.365 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:18:16.365 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:18:16.365 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:18:16.366 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:18:16.366 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:18:16.366 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:18:16.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:18:16.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:18:16.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:18:16.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:18:16.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:18:16.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:18:16.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:18:16.658 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:18:16.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:18:16.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:18:16.659 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:18:16.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=356482285570 2026-02-13T02:18:16.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 356482285570 2026-02-13T02:18:16.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-356482285570' 2026-02-13T02:18:16.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:18:16.732 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:18:16.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=335007449097 2026-02-13T02:18:16.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 335007449097 2026-02-13T02:18:16.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-356482285570 1-335007449097' 2026-02-13T02:18:16.806 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:18:16.806 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-356482285570 2026-02-13T02:18:16.806 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:18:16.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:18:16.808 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-356482285570 2026-02-13T02:18:16.809 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:18:16.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=356482285570 2026-02-13T02:18:16.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 356482285570' 2026-02-13T02:18:16.811 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 356482285570 2026-02-13T02:18:16.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:18:16.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 313532612620 -lt 356482285570 2026-02-13T02:18:16.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:18:17.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:18:17.979 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:18:18.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 356482285571 -lt 356482285570 2026-02-13T02:18:18.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:18:18.144 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-335007449097 2026-02-13T02:18:18.144 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:18:18.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:18:18.146 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-335007449097 2026-02-13T02:18:18.146 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:18:18.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=335007449097 2026-02-13T02:18:18.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 335007449097' 2026-02-13T02:18:18.148 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 335007449097 2026-02-13T02:18:18.148 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:18:18.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 335007449097 -lt 335007449097 2026-02-13T02:18:18.314 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:18:18.315 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:18:18.315 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:18:18.505 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:18:18.505 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:18:18.506 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:18:18.506 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:18:18.506 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:18:18.506 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:18:18.507 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:18:18.507 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:18:18.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:18:18.670 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:18:18.671 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:18:18.671 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:18:18.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:18:18.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:18:18.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:18:18.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-13T02:18:18.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ5 2026-02-13T02:18:18.864 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 5 % 2 2026-02-13T02:18:18.866 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=1 2026-02-13T02:18:18.866 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-13T02:18:18.866 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6513: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2026-02-13T02:18:18.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:18:18.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:18:18.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:18:18.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:18:18.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2026-02-13T02:18:18.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:18:18.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:18:18.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:18:18.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:18:18.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:18:18.868 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:18:18.868 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:18:18.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:18:18.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:18:18.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:18:19.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:18:19.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2026-02-13T02:18:19.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:18:19.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:18:19.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:18:19.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:18:19.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:18:19.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ5 rm-omap key-ROBJ5 2026-02-13T02:18:19.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:18:19.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:18:19.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:18:19.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:18:19.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:18:19.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:18:19.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:18:19.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:18:19.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:18:19.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:18:19.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:18:19.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:18:19.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:18:19.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:18:19.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:18:19.451 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:18:19.451 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:18:19.451 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:18:19.452 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:18:19.452 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:18:19.452 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:18:19.454 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:18:19.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:18:19.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:18:19.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:18:19.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:18:19.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:18:19.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:18:19.455 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:18:19.455 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:18:19.455 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:18:19.455 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-13T02:18:19.455 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:18:19.455 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:18:19.455 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:18:19.455 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:18:19.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:18:19.455 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:18:19.455 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:18:19.455 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:18:19.456 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:18:19.473 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:19.461+0000 7f506a237a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:18:19.475 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:19.461+0000 7f506a237a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:18:19.476 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:19.461+0000 7f506a237a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:18:19.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:18:19.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:18:19.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:18:19.613 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:18:19.613 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:18:19.613 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:18:19.613 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:18:19.613 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:18:19.613 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:18:19.613 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:18:19.779 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:18:19.870 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:19.857+0000 7f506a237a40 -1 Falling back to public interface 2026-02-13T02:18:20.428 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:20.413+0000 7f506a237a40 -1 osd.1 84 log_to_monitors true 2026-02-13T02:18:20.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:18:20.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:18:20.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:18:20.781 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:18:20.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:18:20.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:18:20.947 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 88 up_thru 88 down_at 85 last_clean_interval [78,84) [v2:127.0.0.1:6810/3054053363,v1:127.0.0.1:6811/3054053363] [v2:127.0.0.1:6812/3054053363,v1:127.0.0.1:6813/3054053363] exists,up 65f77df4-d31d-4013-880d-185b2abb9aaf 2026-02-13T02:18:20.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:18:20.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:18:20.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:18:20.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:18:20.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:18:20.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:18:20.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:18:20.948 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:18:20.949 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:18:20.949 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:18:20.949 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:18:20.949 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:18:20.949 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:18:21.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:18:21.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:18:21.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:18:21.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:18:21.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:18:21.063 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:18:21.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:18:21.234 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:18:21.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:18:21.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:18:21.235 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:18:21.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=356482285576 2026-02-13T02:18:21.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 356482285576 2026-02-13T02:18:21.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-356482285576' 2026-02-13T02:18:21.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:18:21.308 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:18:21.381 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=377957122050 2026-02-13T02:18:21.381 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 377957122050 2026-02-13T02:18:21.381 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-356482285576 1-377957122050' 2026-02-13T02:18:21.381 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:18:21.382 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-356482285576 2026-02-13T02:18:21.382 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:18:21.383 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:18:21.384 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-356482285576 2026-02-13T02:18:21.384 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:18:21.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=356482285576 2026-02-13T02:18:21.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 356482285576' 2026-02-13T02:18:21.386 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 356482285576 2026-02-13T02:18:21.386 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:18:21.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 356482285575 -lt 356482285576 2026-02-13T02:18:21.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:18:22.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:18:22.561 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:18:22.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 356482285575 -lt 356482285576 2026-02-13T02:18:22.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:18:23.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:18:23.727 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:18:23.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 356482285578 -lt 356482285576 2026-02-13T02:18:23.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:18:23.886 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-377957122050 2026-02-13T02:18:23.886 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:18:23.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:18:23.888 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-377957122050 2026-02-13T02:18:23.889 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:18:23.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=377957122050 2026-02-13T02:18:23.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 377957122050' 2026-02-13T02:18:23.891 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 377957122050 2026-02-13T02:18:23.891 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:18:24.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 377957122052 -lt 377957122050 2026-02-13T02:18:24.063 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:18:24.064 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:18:24.064 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:18:24.266 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:18:24.266 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:18:24.267 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:18:24.268 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:18:24.268 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:18:24.268 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:18:24.268 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:18:24.268 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:18:24.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:18:24.425 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:18:24.425 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:18:24.426 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:18:24.624 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:18:24.624 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:18:24.624 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:18:24.624 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-13T02:18:24.624 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ6 2026-02-13T02:18:24.624 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 6 % 2 2026-02-13T02:18:24.627 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=0 2026-02-13T02:18:24.627 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-13T02:18:24.627 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6518: TEST_dual_store_replicated_cluster: echo extra 2026-02-13T02:18:24.627 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6519: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2026-02-13T02:18:24.627 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:18:24.627 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:18:24.627 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T02:18:24.627 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:18:24.627 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2026-02-13T02:18:24.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:18:24.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:18:24.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T02:18:24.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:18:24.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T02:18:24.628 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:18:24.628 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:18:24.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:18:24.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:18:24.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:18:24.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:18:24.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2026-02-13T02:18:24.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:18:24.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:18:24.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:18:24.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:18:24.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:18:24.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2026-02-13T02:18:25.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:18:25.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:18:25.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:18:25.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:18:25.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:18:25.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:18:25.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:18:25.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:18:25.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:18:25.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:18:25.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:18:25.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:18:25.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:18:25.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:18:25.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:18:25.241 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:18:25.241 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:18:25.241 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:18:25.241 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:18:25.241 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:18:25.242 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:18:25.242 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:18:25.242 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:18:25.242 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:18:25.242 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:18:25.242 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:18:25.242 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:18:25.242 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:18:25.242 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:18:25.242 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:18:25.242 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:18:25.243 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-13T02:18:25.243 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:18:25.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:18:25.244 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T02:18:25.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:18:25.244 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T02:18:25.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:18:25.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:18:25.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:18:25.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:18:25.261 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:25.249+0000 7f70762cda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:18:25.263 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:25.249+0000 7f70762cda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:18:25.264 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:25.249+0000 7f70762cda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:18:25.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:18:25.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:18:25.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:18:25.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:18:25.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:18:25.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:18:25.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:18:25.408 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:18:25.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:18:25.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:18:25.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:18:25.622 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:25.609+0000 7f70762cda40 -1 Falling back to public interface 2026-02-13T02:18:26.230 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:26.217+0000 7f70762cda40 -1 osd.0 89 log_to_monitors true 2026-02-13T02:18:26.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:18:26.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:18:26.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:18:26.571 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:18:26.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:18:26.572 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:18:26.734 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 93 up_thru 93 down_at 90 last_clean_interval [83,89) [v2:127.0.0.1:6800/2129147529,v1:127.0.0.1:6801/2129147529] [v2:127.0.0.1:6802/2129147529,v1:127.0.0.1:6803/2129147529] exists,up 4f8d5ce6-a345-424d-8fc8-9eb1e4df45a1 2026-02-13T02:18:26.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:18:26.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:18:26.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:18:26.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:18:26.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:18:26.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:18:26.735 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:18:26.735 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:18:26.736 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:18:26.736 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:18:26.736 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:18:26.736 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:18:26.736 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:18:26.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:18:26.854 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:18:26.854 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:18:26.854 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:18:26.854 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:18:26.854 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:18:27.015 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:18:27.015 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:18:27.015 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:18:27.015 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:18:27.015 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:18:27.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=399431958530 2026-02-13T02:18:27.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 399431958530 2026-02-13T02:18:27.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-399431958530' 2026-02-13T02:18:27.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:18:27.088 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:18:27.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=377957122057 2026-02-13T02:18:27.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 377957122057 2026-02-13T02:18:27.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-399431958530 1-377957122057' 2026-02-13T02:18:27.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:18:27.164 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-399431958530 2026-02-13T02:18:27.164 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:18:27.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:18:27.166 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-399431958530 2026-02-13T02:18:27.166 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:18:27.167 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=399431958530 2026-02-13T02:18:27.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 399431958530' 2026-02-13T02:18:27.168 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 399431958530 2026-02-13T02:18:27.168 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:18:27.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 399431958530 2026-02-13T02:18:27.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:18:28.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:18:28.343 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:18:28.510 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 399431958530 2026-02-13T02:18:28.510 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:18:29.511 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:18:29.512 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:18:29.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 399431958532 -lt 399431958530 2026-02-13T02:18:29.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:18:29.668 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-377957122057 2026-02-13T02:18:29.668 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:18:29.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:18:29.670 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-377957122057 2026-02-13T02:18:29.671 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:18:29.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=377957122057 2026-02-13T02:18:29.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 377957122057' 2026-02-13T02:18:29.673 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 377957122057 2026-02-13T02:18:29.673 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:18:29.839 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 377957122059 -lt 377957122057 2026-02-13T02:18:29.839 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:18:29.840 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:18:29.840 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:18:30.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:18:30.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:18:30.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:18:30.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:18:30.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:18:30.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:18:30.048 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:18:30.048 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:18:30.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:18:30.206 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:18:30.206 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:18:30.206 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:18:30.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:18:30.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:18:30.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:18:30.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6520: TEST_dual_store_replicated_cluster: rm td/osd-scrub-repair/extra-val 2026-02-13T02:18:30.401 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-13T02:18:30.401 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ7 2026-02-13T02:18:30.401 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 7 % 2 2026-02-13T02:18:30.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=1 2026-02-13T02:18:30.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-13T02:18:30.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6525: TEST_dual_store_replicated_cluster: echo -n newheader 2026-02-13T02:18:30.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6526: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2026-02-13T02:18:30.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:18:30.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:18:30.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:18:30.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:18:30.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2026-02-13T02:18:30.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:18:30.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:18:30.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:18:30.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:18:30.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:18:30.405 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:18:30.405 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:18:30.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:18:30.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:18:30.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:18:30.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:18:30.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2026-02-13T02:18:30.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:18:30.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:18:30.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:18:30.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:18:30.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:18:30.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2026-02-13T02:18:30.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:18:30.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:18:30.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:18:30.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:18:30.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:18:30.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:18:30.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:18:30.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:18:30.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:18:30.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:18:30.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:18:30.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:18:30.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:18:30.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:18:30.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:18:30.992 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:18:30.992 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:18:30.992 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:18:30.992 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:18:30.992 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:18:30.992 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:18:30.992 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:18:30.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:18:30.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:18:30.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:18:30.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:18:30.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:18:30.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:18:30.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:18:30.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:18:30.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:18:30.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-13T02:18:30.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:18:30.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:18:30.995 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:18:30.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:18:30.995 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:18:30.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:18:30.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:18:30.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:18:30.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:18:31.010 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:30.997+0000 7f3dc07b7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:18:31.012 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:30.997+0000 7f3dc07b7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:18:31.013 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:31.001+0000 7f3dc07b7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:18:31.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:18:31.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:18:31.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:18:31.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:18:31.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:18:31.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:18:31.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:18:31.157 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:18:31.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:18:31.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:18:31.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:18:31.394 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:31.381+0000 7f3dc07b7a40 -1 Falling back to public interface 2026-02-13T02:18:32.003 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:31.989+0000 7f3dc07b7a40 -1 osd.1 94 log_to_monitors true 2026-02-13T02:18:32.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:18:32.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:18:32.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:18:32.323 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:18:32.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:18:32.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:18:32.483 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 98 up_thru 98 down_at 95 last_clean_interval [88,94) [v2:127.0.0.1:6810/3960910375,v1:127.0.0.1:6811/3960910375] [v2:127.0.0.1:6812/3960910375,v1:127.0.0.1:6813/3960910375] exists,up 65f77df4-d31d-4013-880d-185b2abb9aaf 2026-02-13T02:18:32.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:18:32.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:18:32.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:18:32.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:18:32.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:18:32.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:18:32.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:18:32.484 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:18:32.484 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:18:32.484 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:18:32.485 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:18:32.485 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:18:32.485 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:18:32.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:18:32.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:18:32.594 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:18:32.594 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:18:32.594 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:18:32.594 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:18:32.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:18:32.756 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:18:32.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:18:32.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:18:32.756 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:18:32.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=399431958537 2026-02-13T02:18:32.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 399431958537 2026-02-13T02:18:32.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-399431958537' 2026-02-13T02:18:32.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:18:32.832 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:18:32.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=420906795010 2026-02-13T02:18:32.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 420906795010 2026-02-13T02:18:32.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-399431958537 1-420906795010' 2026-02-13T02:18:32.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:18:32.908 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-399431958537 2026-02-13T02:18:32.908 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:18:32.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:18:32.910 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-399431958537 2026-02-13T02:18:32.910 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:18:32.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=399431958537 2026-02-13T02:18:32.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 399431958537' 2026-02-13T02:18:32.912 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 399431958537 2026-02-13T02:18:32.913 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:18:33.074 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 399431958537 -lt 399431958537 2026-02-13T02:18:33.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:18:33.075 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-420906795010 2026-02-13T02:18:33.075 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:18:33.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:18:33.078 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-420906795010 2026-02-13T02:18:33.078 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:18:33.079 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=420906795010 2026-02-13T02:18:33.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 420906795010' 2026-02-13T02:18:33.080 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 420906795010 2026-02-13T02:18:33.080 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:18:33.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 420906795010 -lt 420906795010 2026-02-13T02:18:33.241 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:18:33.242 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:18:33.242 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:18:33.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:18:33.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:18:33.434 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:18:33.434 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:18:33.434 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:18:33.434 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:18:33.434 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:18:33.434 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:18:33.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:18:33.590 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:18:33.591 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:18:33.591 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:18:33.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:18:33.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:18:33.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:18:33.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6527: TEST_dual_store_replicated_cluster: rm td/osd-scrub-repair/hdr 2026-02-13T02:18:33.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-13T02:18:33.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ8 2026-02-13T02:18:33.793 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 8 % 2 2026-02-13T02:18:33.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=0 2026-02-13T02:18:33.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-13T02:18:33.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6531: TEST_dual_store_replicated_cluster: rados --pool csr_pool setxattr ROBJ8 key1-ROBJ8 val1-ROBJ8 2026-02-13T02:18:33.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6532: TEST_dual_store_replicated_cluster: rados --pool csr_pool setxattr ROBJ8 key2-ROBJ8 val2-ROBJ8 2026-02-13T02:18:33.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6535: TEST_dual_store_replicated_cluster: echo -n bad-val 2026-02-13T02:18:33.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6536: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2026-02-13T02:18:33.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:18:33.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:18:33.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T02:18:33.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:18:33.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2026-02-13T02:18:33.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:18:33.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:18:33.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T02:18:33.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:18:33.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T02:18:33.840 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:18:33.840 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:18:33.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:18:33.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:18:33.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:18:34.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:18:34.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2026-02-13T02:18:34.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:18:34.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:18:34.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:18:34.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:18:34.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:18:34.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2026-02-13T02:18:34.442 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:18:34.442 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:18:34.442 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:18:34.442 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:18:34.442 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:18:34.442 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:18:34.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:18:34.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:18:34.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:18:34.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:18:34.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:18:34.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:18:34.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:18:34.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:18:34.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:18:34.443 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:18:34.443 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:18:34.444 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:18:34.444 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:18:34.444 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:18:34.444 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:18:34.446 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:18:34.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:18:34.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:18:34.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:18:34.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:18:34.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:18:34.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:18:34.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:18:34.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:18:34.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:18:34.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-13T02:18:34.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:18:34.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:18:34.447 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T02:18:34.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:18:34.447 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T02:18:34.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:18:34.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:18:34.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:18:34.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:18:34.466 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:34.453+0000 7fba553f7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:18:34.467 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:34.453+0000 7fba553f7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:18:34.468 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:34.457+0000 7fba553f7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:18:34.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:18:34.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:18:34.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:18:34.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:18:34.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:18:34.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:18:34.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:18:34.603 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:18:34.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:18:34.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:18:34.766 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:18:34.906 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:34.893+0000 7fba553f7a40 -1 Falling back to public interface 2026-02-13T02:18:35.511 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:35.497+0000 7fba553f7a40 -1 osd.0 99 log_to_monitors true 2026-02-13T02:18:35.768 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:18:35.768 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:18:35.768 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:18:35.768 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:18:35.768 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:18:35.768 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:18:35.929 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 103 up_thru 103 down_at 100 last_clean_interval [93,99) [v2:127.0.0.1:6800/2618781892,v1:127.0.0.1:6801/2618781892] [v2:127.0.0.1:6802/2618781892,v1:127.0.0.1:6803/2618781892] exists,up 4f8d5ce6-a345-424d-8fc8-9eb1e4df45a1 2026-02-13T02:18:35.929 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:18:35.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:18:35.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:18:35.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:18:35.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:18:35.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:18:35.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:18:35.931 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:18:35.931 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:18:35.931 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:18:35.931 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:18:35.931 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:18:35.931 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:18:36.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:18:36.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:18:36.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:18:36.047 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:18:36.047 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:18:36.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:18:36.206 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:18:36.207 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:18:36.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:18:36.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:18:36.207 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:18:36.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=442381631490 2026-02-13T02:18:36.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 442381631490 2026-02-13T02:18:36.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-442381631490' 2026-02-13T02:18:36.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:18:36.284 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:18:36.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=420906795015 2026-02-13T02:18:36.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 420906795015 2026-02-13T02:18:36.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-442381631490 1-420906795015' 2026-02-13T02:18:36.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:18:36.356 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-442381631490 2026-02-13T02:18:36.356 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:18:36.358 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:18:36.359 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-442381631490 2026-02-13T02:18:36.359 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:18:36.361 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=442381631490 2026-02-13T02:18:36.361 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 442381631490' 2026-02-13T02:18:36.361 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 442381631490 2026-02-13T02:18:36.361 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:18:36.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 399431958538 -lt 442381631490 2026-02-13T02:18:36.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:18:37.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:18:37.520 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:18:37.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 442381631491 -lt 442381631490 2026-02-13T02:18:37.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:18:37.678 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-420906795015 2026-02-13T02:18:37.678 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:18:37.680 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:18:37.681 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-420906795015 2026-02-13T02:18:37.681 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:18:37.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=420906795015 2026-02-13T02:18:37.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 420906795015' 2026-02-13T02:18:37.683 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 420906795015 2026-02-13T02:18:37.683 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:18:37.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 420906795015 -lt 420906795015 2026-02-13T02:18:37.857 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:18:37.857 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:18:37.857 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:18:38.057 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:18:38.057 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:18:38.058 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:18:38.058 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:18:38.058 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:18:38.058 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:18:38.059 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:18:38.059 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:18:38.217 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:18:38.218 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:18:38.218 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:18:38.218 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:18:38.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:18:38.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:18:38.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:18:38.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6537: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2026-02-13T02:18:38.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:18:38.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:18:38.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:18:38.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:18:38.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2026-02-13T02:18:38.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:18:38.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:18:38.410 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:18:38.410 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:18:38.410 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:18:38.410 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:18:38.410 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:18:38.410 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:18:38.410 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:18:38.410 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:18:38.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:18:38.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2026-02-13T02:18:38.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:18:38.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:18:38.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:18:38.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:18:38.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:18:38.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 rm-attr _key2-ROBJ8 2026-02-13T02:18:38.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:18:38.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:18:38.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:18:38.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:18:38.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:18:38.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:18:38.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:18:38.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:18:38.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:18:38.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:18:38.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:18:38.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:18:38.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:18:38.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:18:38.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:18:38.976 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:18:38.976 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:18:38.976 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:18:38.976 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:18:38.976 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:18:38.976 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:18:38.976 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:18:38.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:18:38.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:18:38.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:18:38.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:18:38.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:18:38.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:18:38.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:18:38.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:18:38.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:18:38.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-13T02:18:38.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:18:38.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:18:38.979 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:18:38.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:18:38.979 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:18:38.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:18:38.980 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:18:38.980 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:18:38.980 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:18:38.994 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:38.981+0000 7f03aa19ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:18:38.996 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:38.981+0000 7f03aa19ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:18:38.997 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:38.985+0000 7f03aa19ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:18:39.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:18:39.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:18:39.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:18:39.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:18:39.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:18:39.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:18:39.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:18:39.142 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:18:39.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:18:39.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:18:39.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:18:39.370 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:39.357+0000 7f03aa19ca40 -1 Falling back to public interface 2026-02-13T02:18:39.955 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:39.941+0000 7f03aa19ca40 -1 osd.1 105 log_to_monitors true 2026-02-13T02:18:40.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:18:40.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:18:40.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:18:40.304 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:18:40.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:18:40.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:18:40.463 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 109 up_thru 109 down_at 106 last_clean_interval [98,105) [v2:127.0.0.1:6810/2116321925,v1:127.0.0.1:6811/2116321925] [v2:127.0.0.1:6812/2116321925,v1:127.0.0.1:6813/2116321925] exists,up 65f77df4-d31d-4013-880d-185b2abb9aaf 2026-02-13T02:18:40.463 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:18:40.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:18:40.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:18:40.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:18:40.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:18:40.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:18:40.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:18:40.464 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:18:40.465 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:18:40.465 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:18:40.465 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:18:40.465 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:18:40.465 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:18:40.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:18:40.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:18:40.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:18:40.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:18:40.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:18:40.579 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:18:40.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:18:40.732 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:18:40.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:18:40.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:18:40.732 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:18:40.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=442381631496 2026-02-13T02:18:40.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 442381631496 2026-02-13T02:18:40.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-442381631496' 2026-02-13T02:18:40.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:18:40.805 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:18:40.876 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=468151435266 2026-02-13T02:18:40.876 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 468151435266 2026-02-13T02:18:40.877 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-442381631496 1-468151435266' 2026-02-13T02:18:40.877 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:18:40.877 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-442381631496 2026-02-13T02:18:40.877 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:18:40.878 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:18:40.879 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-442381631496 2026-02-13T02:18:40.879 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:18:40.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=442381631496 2026-02-13T02:18:40.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 442381631496' 2026-02-13T02:18:40.881 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 442381631496 2026-02-13T02:18:40.882 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:18:41.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 442381631493 -lt 442381631496 2026-02-13T02:18:41.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:18:42.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:18:42.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:18:42.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 442381631496 -lt 442381631496 2026-02-13T02:18:42.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:18:42.209 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-468151435266 2026-02-13T02:18:42.209 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:18:42.210 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:18:42.212 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-468151435266 2026-02-13T02:18:42.212 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:18:42.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=468151435266 2026-02-13T02:18:42.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 468151435266' 2026-02-13T02:18:42.214 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 468151435266 2026-02-13T02:18:42.214 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:18:42.383 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 468151435266 -lt 468151435266 2026-02-13T02:18:42.384 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:18:42.385 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:18:42.385 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:18:42.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:18:42.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:18:42.586 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:18:42.586 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:18:42.587 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:18:42.587 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:18:42.587 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:18:42.587 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:18:42.747 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:18:42.748 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:18:42.748 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:18:42.748 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:18:42.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:18:42.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:18:42.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:18:42.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6538: TEST_dual_store_replicated_cluster: echo -n val3-ROBJ8 2026-02-13T02:18:42.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6539: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2026-02-13T02:18:42.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:18:42.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:18:42.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:18:42.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:18:42.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2026-02-13T02:18:42.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:18:42.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:18:42.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:18:42.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:18:42.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:18:42.947 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:18:42.947 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:18:42.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:18:42.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:18:42.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:18:43.255 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:18:43.255 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2026-02-13T02:18:43.255 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:18:43.255 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:18:43.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:18:43.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:18:43.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:18:43.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2026-02-13T02:18:43.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:18:43.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:18:43.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:18:43.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:18:43.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:18:43.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:18:43.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:18:43.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:18:43.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:18:43.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:18:43.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:18:43.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:18:43.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:18:43.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:18:43.536 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:18:43.536 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:18:43.536 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:18:43.536 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:18:43.536 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:18:43.536 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:18:43.536 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:18:43.536 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:18:43.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:18:43.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:18:43.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:18:43.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:18:43.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:18:43.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:18:43.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:18:43.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:18:43.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:18:43.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-13T02:18:43.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:18:43.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:18:43.539 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:18:43.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:18:43.539 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:18:43.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:18:43.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:18:43.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:18:43.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:18:43.554 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:43.541+0000 7f8a4bc29a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:18:43.556 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:43.541+0000 7f8a4bc29a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:18:43.557 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:43.545+0000 7f8a4bc29a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:18:43.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:18:43.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:18:43.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:18:43.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:18:43.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:18:43.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:18:43.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:18:43.701 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:18:43.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:18:43.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:18:43.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:18:43.922 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:43.909+0000 7f8a4bc29a40 -1 Falling back to public interface 2026-02-13T02:18:44.541 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:44.529+0000 7f8a4bc29a40 -1 osd.1 111 log_to_monitors true 2026-02-13T02:18:44.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:18:44.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:18:44.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:18:44.865 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:18:44.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:18:44.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:18:45.024 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 115 up_thru 115 down_at 112 last_clean_interval [109,111) [v2:127.0.0.1:6810/190504646,v1:127.0.0.1:6811/190504646] [v2:127.0.0.1:6812/190504646,v1:127.0.0.1:6813/190504646] exists,up 65f77df4-d31d-4013-880d-185b2abb9aaf 2026-02-13T02:18:45.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:18:45.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:18:45.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:18:45.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:18:45.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:18:45.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:18:45.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:18:45.025 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:18:45.026 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:18:45.026 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:18:45.026 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:18:45.026 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:18:45.026 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:18:45.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:18:45.154 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:18:45.154 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:18:45.154 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:18:45.154 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:18:45.154 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:18:45.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:18:45.317 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:18:45.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:18:45.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:18:45.317 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:18:45.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=442381631501 2026-02-13T02:18:45.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 442381631501 2026-02-13T02:18:45.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-442381631501' 2026-02-13T02:18:45.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:18:45.391 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:18:45.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=493921239042 2026-02-13T02:18:45.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 493921239042 2026-02-13T02:18:45.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-442381631501 1-493921239042' 2026-02-13T02:18:45.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:18:45.468 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-442381631501 2026-02-13T02:18:45.468 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:18:45.469 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:18:45.470 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-442381631501 2026-02-13T02:18:45.470 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:18:45.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=442381631501 2026-02-13T02:18:45.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 442381631501' 2026-02-13T02:18:45.472 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 442381631501 2026-02-13T02:18:45.472 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:18:45.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 442381631500 -lt 442381631501 2026-02-13T02:18:45.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:18:46.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:18:46.650 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:18:46.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 442381631500 -lt 442381631501 2026-02-13T02:18:46.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:18:47.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:18:47.814 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:18:47.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 442381631503 -lt 442381631501 2026-02-13T02:18:47.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:18:47.978 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-493921239042 2026-02-13T02:18:47.979 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:18:47.980 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:18:47.981 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-493921239042 2026-02-13T02:18:47.981 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:18:47.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=493921239042 2026-02-13T02:18:47.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 493921239042' 2026-02-13T02:18:47.983 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 493921239042 2026-02-13T02:18:47.984 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:18:48.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 493921239044 -lt 493921239042 2026-02-13T02:18:48.163 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:18:48.163 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:18:48.163 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:18:48.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:18:48.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:18:48.362 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:18:48.362 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:18:48.362 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:18:48.362 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:18:48.362 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:18:48.362 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:18:48.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:18:48.529 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:18:48.529 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:18:48.529 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:18:48.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:18:48.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:18:48.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:18:48.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6540: TEST_dual_store_replicated_cluster: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2026-02-13T02:18:48.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-13T02:18:48.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ9 2026-02-13T02:18:48.734 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 9 % 2 2026-02-13T02:18:48.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=1 2026-02-13T02:18:48.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-13T02:18:48.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6544: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ9 get-attr _ 2026-02-13T02:18:48.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:18:48.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:18:48.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:18:48.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:18:48.737 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 get-attr _ 2026-02-13T02:18:48.737 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:18:48.737 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:18:48.737 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:18:48.737 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:18:48.737 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:18:48.737 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:18:48.737 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:18:48.737 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:18:48.737 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:18:48.737 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:18:49.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:18:49.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 get-attr _ 2026-02-13T02:18:49.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:18:49.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:18:49.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:18:49.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:18:49.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:18:49.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 get-attr _ 2026-02-13T02:18:49.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:18:49.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:18:49.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:18:49.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:18:49.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:18:49.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:18:49.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:18:49.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:18:49.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:18:49.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:18:49.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:18:49.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:18:49.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:18:49.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:18:49.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:18:49.193 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:18:49.193 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:18:49.193 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:18:49.193 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:18:49.193 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:18:49.193 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:18:49.193 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:18:49.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:18:49.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:18:49.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:18:49.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:18:49.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:18:49.195 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:18:49.195 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:18:49.195 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:18:49.195 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:18:49.195 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-13T02:18:49.195 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:18:49.195 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:18:49.195 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:18:49.195 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:18:49.196 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:18:49.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:18:49.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:18:49.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:18:49.198 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:18:49.217 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:49.205+0000 7f69037d5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:18:49.218 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:49.205+0000 7f69037d5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:18:49.219 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:49.205+0000 7f69037d5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:18:49.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:18:49.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:18:49.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:18:49.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:18:49.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:18:49.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:18:49.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:18:49.364 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:18:49.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:18:49.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:18:49.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:18:49.606 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:49.593+0000 7f69037d5a40 -1 Falling back to public interface 2026-02-13T02:18:50.193 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:50.181+0000 7f69037d5a40 -1 osd.1 116 log_to_monitors true 2026-02-13T02:18:50.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:18:50.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:18:50.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:18:50.533 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:18:50.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:18:50.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:18:50.690 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 120 up_thru 120 down_at 117 last_clean_interval [115,116) [v2:127.0.0.1:6810/2301951598,v1:127.0.0.1:6811/2301951598] [v2:127.0.0.1:6812/2301951598,v1:127.0.0.1:6813/2301951598] exists,up 65f77df4-d31d-4013-880d-185b2abb9aaf 2026-02-13T02:18:50.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:18:50.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:18:50.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:18:50.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:18:50.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:18:50.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:18:50.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:18:50.691 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:18:50.692 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:18:50.692 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:18:50.692 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:18:50.692 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:18:50.692 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:18:50.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:18:50.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:18:50.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:18:50.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:18:50.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:18:50.810 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:18:50.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:18:50.973 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:18:50.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:18:50.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:18:50.973 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:18:51.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=442381631508 2026-02-13T02:18:51.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 442381631508 2026-02-13T02:18:51.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-442381631508' 2026-02-13T02:18:51.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:18:51.046 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:18:51.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=515396075522 2026-02-13T02:18:51.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 515396075522 2026-02-13T02:18:51.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-442381631508 1-515396075522' 2026-02-13T02:18:51.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:18:51.121 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-442381631508 2026-02-13T02:18:51.121 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:18:51.122 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:18:51.124 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-442381631508 2026-02-13T02:18:51.124 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:18:51.125 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=442381631508 2026-02-13T02:18:51.125 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 442381631508' 2026-02-13T02:18:51.125 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 442381631508 2026-02-13T02:18:51.126 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:18:51.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 442381631507 -lt 442381631508 2026-02-13T02:18:51.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:18:52.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:18:52.286 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:18:52.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 442381631507 -lt 442381631508 2026-02-13T02:18:52.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:18:53.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:18:53.452 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:18:53.618 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 442381631510 -lt 442381631508 2026-02-13T02:18:53.618 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:18:53.619 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-515396075522 2026-02-13T02:18:53.619 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:18:53.620 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:18:53.621 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-515396075522 2026-02-13T02:18:53.621 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:18:53.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=515396075522 2026-02-13T02:18:53.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 515396075522' 2026-02-13T02:18:53.623 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 515396075522 2026-02-13T02:18:53.624 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:18:53.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 515396075524 -lt 515396075522 2026-02-13T02:18:53.790 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:18:53.791 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:18:53.791 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:18:53.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:18:53.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:18:53.982 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:18:53.983 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:18:53.983 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:18:53.983 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:18:53.983 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:18:53.983 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:18:54.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:18:54.138 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:18:54.138 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:18:54.138 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:18:54.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:18:54.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:18:54.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:18:54.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6545: TEST_dual_store_replicated_cluster: echo -n D 2026-02-13T02:18:54.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6546: TEST_dual_store_replicated_cluster: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2026-02-13T02:18:54.359 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6547: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2026-02-13T02:18:54.359 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:18:54.359 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:18:54.359 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:18:54.359 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:18:54.359 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2026-02-13T02:18:54.359 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:18:54.359 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:18:54.359 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:18:54.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:18:54.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:18:54.360 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:18:54.360 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:18:54.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:18:54.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:18:54.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:18:54.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:18:54.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2026-02-13T02:18:54.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:18:54.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:18:54.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:18:54.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:18:54.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:18:54.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2026-02-13T02:18:54.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:18:54.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:18:54.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:18:54.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:18:54.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:18:54.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:18:54.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:18:54.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:18:54.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:18:54.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:18:54.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:18:54.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:18:54.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:18:54.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:18:54.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:18:54.964 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:18:54.964 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:18:54.964 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:18:54.964 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:18:54.964 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:18:54.964 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:18:54.964 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:18:54.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:18:54.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:18:54.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:18:54.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:18:54.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:18:54.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:18:54.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:18:54.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:18:54.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:18:54.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-13T02:18:54.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:18:54.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:18:54.968 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:18:54.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:18:54.968 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:18:54.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:18:54.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:18:54.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:18:54.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:18:54.985 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:54.973+0000 7f699c267a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:18:54.987 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:54.973+0000 7f699c267a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:18:54.988 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:54.973+0000 7f699c267a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:18:55.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:18:55.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:18:55.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:18:55.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:18:55.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:18:55.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:18:55.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:18:55.126 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:18:55.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:18:55.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:18:55.291 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:18:55.386 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:55.373+0000 7f699c267a40 -1 Falling back to public interface 2026-02-13T02:18:55.948 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:55.933+0000 7f699c267a40 -1 osd.1 122 log_to_monitors true 2026-02-13T02:18:56.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:18:56.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:18:56.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:18:56.293 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:18:56.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:18:56.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:18:56.453 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 126 up_thru 126 down_at 123 last_clean_interval [120,122) [v2:127.0.0.1:6810/3848905730,v1:127.0.0.1:6811/3848905730] [v2:127.0.0.1:6812/3848905730,v1:127.0.0.1:6813/3848905730] exists,up 65f77df4-d31d-4013-880d-185b2abb9aaf 2026-02-13T02:18:56.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:18:56.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:18:56.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:18:56.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:18:56.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:18:56.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:18:56.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:18:56.454 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:18:56.454 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:18:56.454 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:18:56.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:18:56.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:18:56.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:18:56.574 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:18:56.574 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:18:56.574 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:18:56.574 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:18:56.574 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:18:56.574 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:18:56.735 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:18:56.735 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:18:56.735 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:18:56.735 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:18:56.735 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:18:56.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=442381631515 2026-02-13T02:18:56.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 442381631515 2026-02-13T02:18:56.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-442381631515' 2026-02-13T02:18:56.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:18:56.809 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:18:56.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=541165879298 2026-02-13T02:18:56.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 541165879298 2026-02-13T02:18:56.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-442381631515 1-541165879298' 2026-02-13T02:18:56.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:18:56.880 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-442381631515 2026-02-13T02:18:56.880 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:18:56.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:18:56.882 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-442381631515 2026-02-13T02:18:56.882 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:18:56.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=442381631515 2026-02-13T02:18:56.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 442381631515' 2026-02-13T02:18:56.884 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 442381631515 2026-02-13T02:18:56.885 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:18:57.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 442381631512 -lt 442381631515 2026-02-13T02:18:57.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:18:58.048 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:18:58.048 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:18:58.210 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 442381631515 -lt 442381631515 2026-02-13T02:18:58.210 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:18:58.210 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-541165879298 2026-02-13T02:18:58.211 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:18:58.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:18:58.213 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-541165879298 2026-02-13T02:18:58.213 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:18:58.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=541165879298 2026-02-13T02:18:58.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 541165879298' 2026-02-13T02:18:58.215 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 541165879298 2026-02-13T02:18:58.215 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:18:58.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 541165879298 -lt 541165879298 2026-02-13T02:18:58.389 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:18:58.389 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:18:58.390 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:18:58.584 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:18:58.584 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:18:58.585 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:18:58.585 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:18:58.585 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:18:58.586 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:18:58.586 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:18:58.586 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:18:58.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:18:58.744 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:18:58.744 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:18:58.744 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:18:58.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:18:58.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:18:58.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:18:58.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6548: TEST_dual_store_replicated_cluster: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2026-02-13T02:18:58.935 INFO:tasks.workunit.client.0.trial112.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2026-02-13T02:18:58.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-13T02:18:58.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ10 2026-02-13T02:18:58.936 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 10 % 2 2026-02-13T02:18:58.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=0 2026-02-13T02:18:58.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-13T02:18:58.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-13T02:18:58.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ11 2026-02-13T02:18:58.937 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 11 % 2 2026-02-13T02:18:58.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=1 2026-02-13T02:18:58.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-13T02:18:58.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-13T02:18:58.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ12 2026-02-13T02:18:58.940 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 12 % 2 2026-02-13T02:18:58.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=0 2026-02-13T02:18:58.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-13T02:18:58.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-13T02:18:58.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ13 2026-02-13T02:18:58.942 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 13 % 2 2026-02-13T02:18:58.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=1 2026-02-13T02:18:58.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-13T02:18:58.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-13T02:18:58.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ14 2026-02-13T02:18:58.943 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 14 % 2 2026-02-13T02:18:58.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=0 2026-02-13T02:18:58.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-13T02:18:58.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6557: TEST_dual_store_replicated_cluster: echo -n bad-val 2026-02-13T02:18:58.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6558: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2026-02-13T02:18:58.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:18:58.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:18:58.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T02:18:58.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:18:58.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2026-02-13T02:18:58.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:18:58.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:18:58.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T02:18:58.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:18:58.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T02:18:58.947 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:18:58.947 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:18:58.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:18:58.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:18:58.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:18:59.255 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:18:59.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2026-02-13T02:18:59.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:18:59.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:18:59.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:18:59.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:18:59.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:18:59.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2026-02-13T02:18:59.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:18:59.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:18:59.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:18:59.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:18:59.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:18:59.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:18:59.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:18:59.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:18:59.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:18:59.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:18:59.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:18:59.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:18:59.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:18:59.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:18:59.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:18:59.562 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:18:59.562 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:18:59.562 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:18:59.563 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:18:59.563 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:18:59.563 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:18:59.563 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:18:59.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:18:59.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:18:59.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:18:59.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:18:59.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:18:59.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:18:59.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:18:59.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:18:59.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:18:59.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-13T02:18:59.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:18:59.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:18:59.565 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T02:18:59.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:18:59.565 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T02:18:59.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:18:59.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:18:59.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:18:59.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:18:59.582 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:59.569+0000 7f08c3efca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:18:59.584 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:59.573+0000 7f08c3efca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:18:59.585 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:59.573+0000 7f08c3efca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:18:59.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:18:59.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:18:59.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:18:59.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:18:59.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:18:59.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:18:59.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:18:59.727 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:18:59.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:18:59.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:18:59.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:18:59.998 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:18:59.985+0000 7f08c3efca40 -1 Falling back to public interface 2026-02-13T02:19:00.611 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:00.597+0000 7f08c3efca40 -1 osd.0 128 log_to_monitors true 2026-02-13T02:19:00.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:19:00.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:19:00.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:19:00.897 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:19:00.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:19:00.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:19:01.059 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 132 up_thru 132 down_at 129 last_clean_interval [103,128) [v2:127.0.0.1:6800/801974236,v1:127.0.0.1:6801/801974236] [v2:127.0.0.1:6802/801974236,v1:127.0.0.1:6803/801974236] exists,up 4f8d5ce6-a345-424d-8fc8-9eb1e4df45a1 2026-02-13T02:19:01.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:19:01.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:19:01.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:19:01.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:19:01.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:19:01.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:19:01.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:19:01.060 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:19:01.061 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:19:01.061 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:19:01.061 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:19:01.061 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:19:01.062 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:19:01.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:19:01.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:19:01.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:19:01.178 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:19:01.178 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:19:01.178 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:19:01.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:19:01.337 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:19:01.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:19:01.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:19:01.338 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:19:01.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=566935683074 2026-02-13T02:19:01.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 566935683074 2026-02-13T02:19:01.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-566935683074' 2026-02-13T02:19:01.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:19:01.412 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:19:01.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=541165879304 2026-02-13T02:19:01.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 541165879304 2026-02-13T02:19:01.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-566935683074 1-541165879304' 2026-02-13T02:19:01.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:19:01.485 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-566935683074 2026-02-13T02:19:01.485 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:19:01.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:19:01.487 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-566935683074 2026-02-13T02:19:01.487 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:19:01.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=566935683074 2026-02-13T02:19:01.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 566935683074' 2026-02-13T02:19:01.489 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 566935683074 2026-02-13T02:19:01.489 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:19:01.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 566935683074 2026-02-13T02:19:01.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:19:02.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:19:02.653 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:19:02.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 566935683074 2026-02-13T02:19:02.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:19:03.816 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:19:03.816 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:19:03.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 566935683076 -lt 566935683074 2026-02-13T02:19:03.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:19:03.980 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-541165879304 2026-02-13T02:19:03.980 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:19:03.981 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:19:03.983 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-541165879304 2026-02-13T02:19:03.983 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:19:03.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=541165879304 2026-02-13T02:19:03.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 541165879304' 2026-02-13T02:19:03.984 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 541165879304 2026-02-13T02:19:03.985 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:19:04.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 541165879305 -lt 541165879304 2026-02-13T02:19:04.150 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:19:04.150 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:19:04.151 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:19:04.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:19:04.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:19:04.345 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:19:04.345 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:19:04.345 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:19:04.345 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:19:04.346 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:19:04.346 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:19:04.502 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:19:04.503 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:19:04.503 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:19:04.503 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:19:04.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:19:04.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:19:04.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:19:04.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6559: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2026-02-13T02:19:04.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:19:04.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:19:04.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:19:04.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:19:04.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2026-02-13T02:19:04.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:19:04.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:19:04.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:19:04.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:19:04.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:19:04.702 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:19:04.702 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:19:04.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:19:04.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:19:04.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:19:05.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:19:05.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2026-02-13T02:19:05.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:19:05.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:19:05.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:19:05.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:19:05.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:19:05.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ14 rm-attr _ 2026-02-13T02:19:05.291 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:19:05.291 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:19:05.291 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:19:05.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:19:05.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:19:05.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:19:05.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:19:05.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:19:05.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:19:05.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:19:05.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:19:05.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:19:05.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:19:05.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:19:05.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:19:05.293 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:19:05.293 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:19:05.293 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:19:05.293 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:19:05.293 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:19:05.293 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:19:05.293 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:19:05.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:19:05.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:19:05.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:19:05.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:19:05.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:19:05.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:19:05.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:19:05.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:19:05.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:19:05.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-13T02:19:05.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:19:05.295 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:19:05.296 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:19:05.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:19:05.296 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:19:05.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:19:05.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:19:05.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:19:05.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:19:05.311 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:05.297+0000 7f1f24202a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:19:05.313 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:05.301+0000 7f1f24202a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:19:05.314 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:05.301+0000 7f1f24202a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:19:05.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:19:05.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:19:05.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:19:05.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:19:05.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:19:05.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:19:05.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:19:05.459 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:19:05.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:19:05.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:19:05.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:19:05.662 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:05.649+0000 7f1f24202a40 -1 Falling back to public interface 2026-02-13T02:19:06.300 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:06.289+0000 7f1f24202a40 -1 osd.1 133 log_to_monitors true 2026-02-13T02:19:06.619 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:19:06.620 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:19:06.620 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:19:06.620 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:19:06.620 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:19:06.620 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:19:06.779 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 137 up_thru 137 down_at 134 last_clean_interval [126,133) [v2:127.0.0.1:6810/774809732,v1:127.0.0.1:6811/774809732] [v2:127.0.0.1:6812/774809732,v1:127.0.0.1:6813/774809732] exists,up 65f77df4-d31d-4013-880d-185b2abb9aaf 2026-02-13T02:19:06.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:19:06.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:19:06.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:19:06.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:19:06.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:19:06.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:19:06.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:19:06.780 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:19:06.781 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:19:06.781 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:19:06.781 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:19:06.781 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:19:06.782 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:19:06.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:19:06.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:19:06.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:19:06.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:19:06.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:19:06.903 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:19:07.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:19:07.063 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:19:07.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:19:07.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:19:07.063 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:19:07.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=566935683081 2026-02-13T02:19:07.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 566935683081 2026-02-13T02:19:07.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-566935683081' 2026-02-13T02:19:07.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:19:07.136 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:19:07.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=588410519554 2026-02-13T02:19:07.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 588410519554 2026-02-13T02:19:07.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-566935683081 1-588410519554' 2026-02-13T02:19:07.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:19:07.210 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-566935683081 2026-02-13T02:19:07.210 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:19:07.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:19:07.212 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-566935683081 2026-02-13T02:19:07.212 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:19:07.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=566935683081 2026-02-13T02:19:07.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 566935683081' 2026-02-13T02:19:07.214 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 566935683081 2026-02-13T02:19:07.214 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:19:07.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 566935683080 -lt 566935683081 2026-02-13T02:19:07.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:19:08.382 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:19:08.382 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:19:08.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 566935683080 -lt 566935683081 2026-02-13T02:19:08.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:19:09.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:19:09.545 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:19:09.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 566935683083 -lt 566935683081 2026-02-13T02:19:09.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:19:09.709 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-588410519554 2026-02-13T02:19:09.709 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:19:09.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:19:09.712 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-588410519554 2026-02-13T02:19:09.712 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:19:09.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=588410519554 2026-02-13T02:19:09.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 588410519554' 2026-02-13T02:19:09.714 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 588410519554 2026-02-13T02:19:09.714 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:19:09.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 588410519556 -lt 588410519554 2026-02-13T02:19:09.886 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:19:09.887 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:19:09.887 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:19:10.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:19:10.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:19:10.088 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:19:10.088 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:19:10.088 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:19:10.088 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:19:10.089 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:19:10.089 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:19:10.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:19:10.248 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:19:10.248 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:19:10.248 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:19:10.441 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:19:10.441 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:19:10.441 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:19:10.441 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6560: TEST_dual_store_replicated_cluster: rm td/osd-scrub-repair/bad-val 2026-02-13T02:19:10.442 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-13T02:19:10.442 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ15 2026-02-13T02:19:10.442 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 15 % 2 2026-02-13T02:19:10.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=1 2026-02-13T02:19:10.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-13T02:19:10.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6564: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2026-02-13T02:19:10.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:19:10.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:19:10.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:19:10.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:19:10.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2026-02-13T02:19:10.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:19:10.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:19:10.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:19:10.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:19:10.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:19:10.446 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:19:10.446 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:19:10.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:19:10.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:19:10.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:19:10.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:19:10.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2026-02-13T02:19:10.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:19:10.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:19:10.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:19:10.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:19:10.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:19:10.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ15 rm-attr _ 2026-02-13T02:19:10.936 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2026-02-13T02:19:11.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:19:11.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:19:11.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:19:11.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:19:11.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:19:11.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:19:11.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:19:11.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:19:11.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:19:11.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:19:11.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:19:11.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:19:11.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:19:11.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:19:11.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:19:11.029 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:19:11.030 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:19:11.030 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:19:11.030 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:19:11.030 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:19:11.030 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:19:11.030 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:19:11.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:19:11.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:19:11.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:19:11.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:19:11.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:19:11.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:19:11.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:19:11.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:19:11.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:19:11.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-13T02:19:11.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:19:11.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:19:11.031 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:19:11.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:19:11.033 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:19:11.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:19:11.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:19:11.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:19:11.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:19:11.047 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:11.032+0000 7f3b8ad11a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:19:11.049 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:11.036+0000 7f3b8ad11a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:19:11.050 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:11.036+0000 7f3b8ad11a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:19:11.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:19:11.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:19:11.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:19:11.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:19:11.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:19:11.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:19:11.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:19:11.188 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:19:11.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:19:11.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:19:11.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:19:11.438 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:11.424+0000 7f3b8ad11a40 -1 Falling back to public interface 2026-02-13T02:19:12.076 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:12.064+0000 7f3b8ad11a40 -1 osd.1 139 log_to_monitors true 2026-02-13T02:19:12.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:19:12.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:19:12.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:19:12.354 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:19:12.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:19:12.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:19:12.518 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 143 up_thru 143 down_at 140 last_clean_interval [137,139) [v2:127.0.0.1:6810/682853262,v1:127.0.0.1:6811/682853262] [v2:127.0.0.1:6812/682853262,v1:127.0.0.1:6813/682853262] exists,up 65f77df4-d31d-4013-880d-185b2abb9aaf 2026-02-13T02:19:12.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:19:12.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:19:12.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:19:12.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:19:12.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:19:12.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:19:12.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:19:12.519 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:19:12.520 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:19:12.520 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:19:12.520 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:19:12.520 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:19:12.520 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:19:12.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:19:12.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:19:12.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:19:12.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:19:12.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:19:12.637 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:19:12.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:19:12.798 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:19:12.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:19:12.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:19:12.799 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:19:12.877 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=566935683088 2026-02-13T02:19:12.877 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 566935683088 2026-02-13T02:19:12.877 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-566935683088' 2026-02-13T02:19:12.878 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:19:12.878 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:19:12.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=614180323330 2026-02-13T02:19:12.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 614180323330 2026-02-13T02:19:12.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-566935683088 1-614180323330' 2026-02-13T02:19:12.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:19:12.952 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-566935683088 2026-02-13T02:19:12.952 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:19:12.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:19:12.954 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-566935683088 2026-02-13T02:19:12.954 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:19:12.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=566935683088 2026-02-13T02:19:12.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 566935683088' 2026-02-13T02:19:12.956 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 566935683088 2026-02-13T02:19:12.957 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:19:13.121 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 566935683088 -lt 566935683088 2026-02-13T02:19:13.121 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:19:13.122 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-614180323330 2026-02-13T02:19:13.122 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:19:13.123 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:19:13.124 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-614180323330 2026-02-13T02:19:13.124 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:19:13.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=614180323330 2026-02-13T02:19:13.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 614180323330' 2026-02-13T02:19:13.126 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 614180323330 2026-02-13T02:19:13.126 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:19:13.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 614180323330 -lt 614180323330 2026-02-13T02:19:13.293 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:19:13.294 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:19:13.294 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:19:13.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:19:13.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:19:13.494 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:19:13.494 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:19:13.494 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:19:13.494 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:19:13.494 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:19:13.494 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:19:13.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:19:13.660 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:19:13.660 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:19:13.660 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:19:13.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:19:13.857 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:19:13.857 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:19:13.857 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-13T02:19:13.857 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ16 2026-02-13T02:19:13.857 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 16 % 2 2026-02-13T02:19:13.859 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=0 2026-02-13T02:19:13.859 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-13T02:19:13.859 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6568: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2026-02-13T02:19:13.859 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:19:13.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:19:13.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T02:19:13.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:19:13.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2026-02-13T02:19:13.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:19:13.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:19:13.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T02:19:13.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:19:13.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T02:19:13.860 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:19:13.860 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:19:13.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:19:13.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:19:13.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:19:14.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:19:14.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2026-02-13T02:19:14.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:19:14.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:19:14.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:19:14.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:19:14.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:19:14.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ16 rm-attr snapset 2026-02-13T02:19:14.336 INFO:tasks.workunit.client.0.trial112.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2026-02-13T02:19:14.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:19:14.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:19:14.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:19:14.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:19:14.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:19:14.455 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:19:14.455 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:19:14.455 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:19:14.455 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:19:14.455 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:19:14.455 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:19:14.455 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:19:14.455 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:19:14.455 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:19:14.455 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:19:14.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:19:14.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:19:14.456 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:19:14.456 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:19:14.456 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:19:14.456 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:19:14.458 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:19:14.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:19:14.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:19:14.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:19:14.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:19:14.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:19:14.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:19:14.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:19:14.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:19:14.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:19:14.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-13T02:19:14.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:19:14.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:19:14.459 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T02:19:14.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:19:14.459 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T02:19:14.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:19:14.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:19:14.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:19:14.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:19:14.477 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:14.464+0000 7f02ee4bfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:19:14.479 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:14.464+0000 7f02ee4bfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:19:14.480 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:14.464+0000 7f02ee4bfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:19:14.621 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:19:14.621 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:19:14.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:19:14.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:19:14.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:19:14.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:19:14.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:19:14.622 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:19:14.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:19:14.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:19:14.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:19:14.874 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:14.860+0000 7f02ee4bfa40 -1 Falling back to public interface 2026-02-13T02:19:15.496 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:15.484+0000 7f02ee4bfa40 -1 osd.0 145 log_to_monitors true 2026-02-13T02:19:15.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:19:15.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:19:15.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:19:15.788 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:19:15.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:19:15.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:19:15.938 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 149 up_thru 149 down_at 146 last_clean_interval [132,145) [v2:127.0.0.1:6800/92662311,v1:127.0.0.1:6801/92662311] [v2:127.0.0.1:6802/92662311,v1:127.0.0.1:6803/92662311] exists,up 4f8d5ce6-a345-424d-8fc8-9eb1e4df45a1 2026-02-13T02:19:15.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:19:15.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:19:15.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:19:15.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:19:15.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:19:15.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:19:15.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:19:15.939 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:19:15.940 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:19:15.940 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:19:15.940 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:19:15.940 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:19:15.940 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:19:16.058 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:19:16.058 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:19:16.058 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:19:16.058 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:19:16.058 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:19:16.058 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:19:16.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:19:16.215 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:19:16.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:19:16.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:19:16.215 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:19:16.289 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=639950127106 2026-02-13T02:19:16.290 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 639950127106 2026-02-13T02:19:16.290 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-639950127106' 2026-02-13T02:19:16.290 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:19:16.290 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:19:16.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=614180323335 2026-02-13T02:19:16.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 614180323335 2026-02-13T02:19:16.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-639950127106 1-614180323335' 2026-02-13T02:19:16.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:19:16.364 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-639950127106 2026-02-13T02:19:16.364 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:19:16.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:19:16.366 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-639950127106 2026-02-13T02:19:16.366 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:19:16.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=639950127106 2026-02-13T02:19:16.368 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 639950127106' 2026-02-13T02:19:16.368 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 639950127106 2026-02-13T02:19:16.368 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:19:16.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 566935683089 -lt 639950127106 2026-02-13T02:19:16.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:19:17.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:19:17.541 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:19:17.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 639950127107 -lt 639950127106 2026-02-13T02:19:17.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:19:17.709 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-614180323335 2026-02-13T02:19:17.709 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:19:17.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:19:17.712 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-614180323335 2026-02-13T02:19:17.712 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:19:17.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=614180323335 2026-02-13T02:19:17.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 614180323335' 2026-02-13T02:19:17.714 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 614180323335 2026-02-13T02:19:17.714 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:19:17.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 614180323335 -lt 614180323335 2026-02-13T02:19:17.885 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:19:17.885 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:19:17.885 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:19:18.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:19:18.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:19:18.079 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:19:18.079 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:19:18.079 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:19:18.080 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:19:18.080 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:19:18.080 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:19:18.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:19:18.244 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:19:18.245 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:19:18.245 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:19:18.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:19:18.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:19:18.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:19:18.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6569: TEST_dual_store_replicated_cluster: echo -n bad-val 2026-02-13T02:19:18.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6570: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2026-02-13T02:19:18.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:19:18.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:19:18.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:19:18.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:19:18.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2026-02-13T02:19:18.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:19:18.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:19:18.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:19:18.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:19:18.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:19:18.447 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:19:18.447 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:19:18.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:19:18.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:19:18.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:19:18.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:19:18.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2026-02-13T02:19:18.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:19:18.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:19:18.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:19:18.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:19:18.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:19:18.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2026-02-13T02:19:18.948 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2026-02-13T02:19:18.948 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2026-02-13T02:19:19.054 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:19:19.055 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:19:19.055 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:19:19.055 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:19:19.055 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:19:19.055 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:19:19.055 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:19:19.055 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:19:19.055 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:19:19.055 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:19:19.055 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:19:19.056 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:19:19.056 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:19:19.056 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:19:19.056 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:19:19.056 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:19:19.056 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:19:19.056 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:19:19.056 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:19:19.056 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:19:19.056 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:19:19.056 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:19:19.058 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:19:19.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:19:19.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:19:19.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:19:19.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:19:19.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:19:19.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:19:19.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:19:19.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:19:19.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-13T02:19:19.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:19:19.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:19:19.060 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:19:19.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:19:19.060 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:19:19.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:19:19.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:19:19.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:19:19.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:19:19.077 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:19.064+0000 7f3210a8fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:19:19.078 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:19.064+0000 7f3210a8fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:19:19.079 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:19.064+0000 7f3210a8fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:19:19.223 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:19:19.223 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:19:19.223 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:19:19.223 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:19:19.223 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:19:19.223 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:19:19.223 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:19:19.224 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:19:19.224 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:19:19.224 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:19:19.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:19:19.510 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:19.496+0000 7f3210a8fa40 -1 Falling back to public interface 2026-02-13T02:19:20.135 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:20.120+0000 7f3210a8fa40 -1 osd.1 150 log_to_monitors true 2026-02-13T02:19:20.385 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:19:20.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:19:20.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:19:20.386 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:19:20.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:19:20.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:19:20.540 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 154 up_thru 154 down_at 151 last_clean_interval [143,150) [v2:127.0.0.1:6810/4155143753,v1:127.0.0.1:6811/4155143753] [v2:127.0.0.1:6812/4155143753,v1:127.0.0.1:6813/4155143753] exists,up 65f77df4-d31d-4013-880d-185b2abb9aaf 2026-02-13T02:19:20.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:19:20.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:19:20.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:19:20.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:19:20.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:19:20.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:19:20.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:19:20.541 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:19:20.542 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:19:20.542 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:19:20.542 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:19:20.542 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:19:20.542 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:19:20.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:19:20.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:19:20.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:19:20.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:19:20.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:19:20.658 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:19:20.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:19:20.822 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:19:20.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:19:20.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:19:20.822 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:19:20.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=639950127112 2026-02-13T02:19:20.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 639950127112 2026-02-13T02:19:20.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-639950127112' 2026-02-13T02:19:20.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:19:20.897 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:19:20.971 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=661424963586 2026-02-13T02:19:20.971 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 661424963586 2026-02-13T02:19:20.971 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-639950127112 1-661424963586' 2026-02-13T02:19:20.971 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:19:20.972 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-639950127112 2026-02-13T02:19:20.972 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:19:20.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:19:20.974 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-639950127112 2026-02-13T02:19:20.974 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:19:20.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=639950127112 2026-02-13T02:19:20.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 639950127112' 2026-02-13T02:19:20.976 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 639950127112 2026-02-13T02:19:20.976 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:19:21.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 639950127112 -lt 639950127112 2026-02-13T02:19:21.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:19:21.137 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-661424963586 2026-02-13T02:19:21.138 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:19:21.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:19:21.140 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-661424963586 2026-02-13T02:19:21.140 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:19:21.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=661424963586 2026-02-13T02:19:21.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 661424963586' 2026-02-13T02:19:21.142 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 661424963586 2026-02-13T02:19:21.142 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:19:21.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 661424963586 -lt 661424963586 2026-02-13T02:19:21.315 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:19:21.315 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:19:21.315 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:19:21.508 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:19:21.508 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:19:21.509 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:19:21.509 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:19:21.509 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:19:21.509 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:19:21.509 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:19:21.509 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:19:21.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:19:21.667 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:19:21.667 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:19:21.667 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:19:21.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:19:21.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:19:21.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:19:21.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-13T02:19:21.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ17 2026-02-13T02:19:21.858 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 17 % 2 2026-02-13T02:19:21.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=1 2026-02-13T02:19:21.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-13T02:19:21.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6575: TEST_dual_store_replicated_cluster: local payload=ROBJ17 2026-02-13T02:19:21.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6576: TEST_dual_store_replicated_cluster: echo ROBJ17 2026-02-13T02:19:21.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6577: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-13T02:19:21.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:19:21.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:19:21.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T02:19:21.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:19:21.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-13T02:19:21.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:19:21.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:19:21.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T02:19:21.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:19:21.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T02:19:21.862 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:19:21.862 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:19:21.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:19:21.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:19:21.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:19:22.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:19:22.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-13T02:19:22.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:19:22.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:19:22.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:19:22.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:19:22.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:19:22.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-13T02:19:22.333 INFO:tasks.workunit.client.0.trial112.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2026-02-13T02:19:22.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:19:22.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:19:22.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:19:22.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:19:22.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:19:22.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:19:22.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:19:22.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:19:22.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:19:22.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:19:22.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:19:22.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:19:22.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:19:22.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:19:22.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:19:22.421 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:19:22.421 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:19:22.421 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:19:22.421 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:19:22.422 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:19:22.422 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:19:22.422 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:19:22.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:19:22.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:19:22.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:19:22.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:19:22.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:19:22.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:19:22.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:19:22.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:19:22.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:19:22.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-13T02:19:22.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:19:22.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:19:22.423 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T02:19:22.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:19:22.424 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T02:19:22.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:19:22.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:19:22.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:19:22.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:19:22.445 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:22.432+0000 7f7fce10aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:19:22.447 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:22.432+0000 7f7fce10aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:19:22.448 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:22.432+0000 7f7fce10aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:19:22.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:19:22.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:19:22.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:19:22.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:19:22.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:19:22.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:19:22.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:19:22.587 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:19:22.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:19:22.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:19:22.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:19:22.814 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:22.800+0000 7f7fce10aa40 -1 Falling back to public interface 2026-02-13T02:19:23.413 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:23.400+0000 7f7fce10aa40 -1 osd.0 155 log_to_monitors true 2026-02-13T02:19:23.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:19:23.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:19:23.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:19:23.754 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:19:23.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:19:23.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:19:23.919 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 159 up_thru 159 down_at 156 last_clean_interval [149,155) [v2:127.0.0.1:6800/15317333,v1:127.0.0.1:6801/15317333] [v2:127.0.0.1:6802/15317333,v1:127.0.0.1:6803/15317333] exists,up 4f8d5ce6-a345-424d-8fc8-9eb1e4df45a1 2026-02-13T02:19:23.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:19:23.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:19:23.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:19:23.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:19:23.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:19:23.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:19:23.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:19:23.921 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:19:23.921 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:19:23.921 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:19:23.921 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:19:23.921 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:19:23.921 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:19:24.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:19:24.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:19:24.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:19:24.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:19:24.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:19:24.035 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:19:24.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:19:24.193 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:19:24.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:19:24.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:19:24.194 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:19:24.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=682899800066 2026-02-13T02:19:24.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 682899800066 2026-02-13T02:19:24.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-682899800066' 2026-02-13T02:19:24.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:19:24.269 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:19:24.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=661424963591 2026-02-13T02:19:24.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 661424963591 2026-02-13T02:19:24.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-682899800066 1-661424963591' 2026-02-13T02:19:24.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:19:24.341 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-682899800066 2026-02-13T02:19:24.341 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:19:24.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:19:24.343 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-682899800066 2026-02-13T02:19:24.343 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:19:24.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=682899800066 2026-02-13T02:19:24.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 682899800066' 2026-02-13T02:19:24.345 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 682899800066 2026-02-13T02:19:24.345 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:19:24.509 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 639950127113 -lt 682899800066 2026-02-13T02:19:24.509 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:19:25.510 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:19:25.511 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:19:25.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 682899800067 -lt 682899800066 2026-02-13T02:19:25.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:19:25.670 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-661424963591 2026-02-13T02:19:25.671 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:19:25.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:19:25.673 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-661424963591 2026-02-13T02:19:25.673 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:19:25.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=661424963591 2026-02-13T02:19:25.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 661424963591' 2026-02-13T02:19:25.675 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 661424963591 2026-02-13T02:19:25.675 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:19:25.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 661424963591 -lt 661424963591 2026-02-13T02:19:25.840 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:19:25.841 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:19:25.841 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:19:26.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:19:26.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:19:26.040 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:19:26.040 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:19:26.040 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:19:26.041 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:19:26.041 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:19:26.041 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:19:26.198 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:19:26.199 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:19:26.199 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:19:26.199 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:19:26.395 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:19:26.395 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:19:26.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:19:26.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6578: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-13T02:19:26.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:19:26.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:19:26.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:19:26.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:19:26.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-13T02:19:26.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:19:26.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:19:26.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:19:26.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:19:26.397 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:19:26.397 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:19:26.397 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:19:26.397 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:19:26.397 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:19:26.397 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:19:26.704 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:19:26.704 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-13T02:19:26.705 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:19:26.705 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:19:26.705 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:19:26.705 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:19:26.705 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:19:26.705 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-13T02:19:26.889 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2026-02-13T02:19:26.889 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2026-02-13T02:19:26.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:19:27.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:19:27.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:19:27.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:19:27.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:19:27.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:19:27.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:19:27.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:19:27.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:19:27.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:19:27.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:19:27.001 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:19:27.001 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:19:27.001 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:19:27.001 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:19:27.001 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:19:27.001 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:19:27.001 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:19:27.001 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:19:27.001 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:19:27.001 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:19:27.001 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:19:27.001 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:19:27.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:19:27.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:19:27.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:19:27.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:19:27.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:19:27.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:19:27.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:19:27.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:19:27.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-13T02:19:27.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:19:27.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:19:27.004 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:19:27.004 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:19:27.004 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:19:27.004 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:19:27.004 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:19:27.004 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:19:27.004 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:19:27.021 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:27.008+0000 7fa334cbda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:19:27.022 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:27.008+0000 7fa334cbda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:19:27.023 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:27.008+0000 7fa334cbda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:19:27.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:19:27.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:19:27.167 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:19:27.167 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:19:27.167 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:19:27.167 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:19:27.167 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:19:27.167 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:19:27.167 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:19:27.167 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:19:27.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:19:27.454 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:27.440+0000 7fa334cbda40 -1 Falling back to public interface 2026-02-13T02:19:28.109 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:28.096+0000 7fa334cbda40 -1 osd.1 161 log_to_monitors true 2026-02-13T02:19:28.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:19:28.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:19:28.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:19:28.330 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:19:28.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:19:28.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:19:28.490 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 165 up_thru 165 down_at 162 last_clean_interval [154,161) [v2:127.0.0.1:6810/3556299954,v1:127.0.0.1:6811/3556299954] [v2:127.0.0.1:6812/3556299954,v1:127.0.0.1:6813/3556299954] exists,up 65f77df4-d31d-4013-880d-185b2abb9aaf 2026-02-13T02:19:28.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:19:28.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:19:28.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:19:28.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:19:28.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:19:28.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:19:28.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:19:28.491 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:19:28.492 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:19:28.492 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:19:28.492 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:19:28.492 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:19:28.492 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:19:28.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:19:28.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:19:28.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:19:28.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:19:28.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:19:28.609 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:19:28.769 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:19:28.770 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:19:28.770 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:19:28.770 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:19:28.770 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:19:28.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=682899800072 2026-02-13T02:19:28.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 682899800072 2026-02-13T02:19:28.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-682899800072' 2026-02-13T02:19:28.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:19:28.844 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:19:28.914 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=708669603842 2026-02-13T02:19:28.914 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 708669603842 2026-02-13T02:19:28.914 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-682899800072 1-708669603842' 2026-02-13T02:19:28.915 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:19:28.915 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-682899800072 2026-02-13T02:19:28.915 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:19:28.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:19:28.918 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-682899800072 2026-02-13T02:19:28.918 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:19:28.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=682899800072 2026-02-13T02:19:28.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 682899800072' 2026-02-13T02:19:28.920 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 682899800072 2026-02-13T02:19:28.920 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:19:29.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 682899800072 -lt 682899800072 2026-02-13T02:19:29.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:19:29.081 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-708669603842 2026-02-13T02:19:29.081 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:19:29.082 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:19:29.083 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-708669603842 2026-02-13T02:19:29.084 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:19:29.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=708669603842 2026-02-13T02:19:29.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 708669603842' 2026-02-13T02:19:29.085 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 708669603842 2026-02-13T02:19:29.086 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:19:29.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 708669603842 -lt 708669603842 2026-02-13T02:19:29.250 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:19:29.250 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:19:29.250 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:19:29.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:19:29.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:19:29.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:19:29.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:19:29.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:19:29.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:19:29.456 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:19:29.456 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:19:29.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:19:29.617 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:19:29.618 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:19:29.618 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:19:29.816 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:19:29.816 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:19:29.816 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:19:29.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-13T02:19:29.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ18 2026-02-13T02:19:29.817 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 18 % 2 2026-02-13T02:19:29.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=0 2026-02-13T02:19:29.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-13T02:19:29.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6583: TEST_dual_store_replicated_cluster: local payload=ROBJ18 2026-02-13T02:19:29.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6584: TEST_dual_store_replicated_cluster: echo ROBJ18 2026-02-13T02:19:29.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6585: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-13T02:19:29.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:19:29.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:19:29.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T02:19:29.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:19:29.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-13T02:19:29.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:19:29.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:19:29.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T02:19:29.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:19:29.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T02:19:29.820 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:19:29.820 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:19:29.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:19:29.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:19:29.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:19:30.129 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:19:30.129 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-13T02:19:30.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:19:30.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:19:30.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:19:30.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:19:30.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:19:30.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-13T02:19:30.321 INFO:tasks.workunit.client.0.trial112.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2026-02-13T02:19:30.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:19:30.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:19:30.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:19:30.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:19:30.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:19:30.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:19:30.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:19:30.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:19:30.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:19:30.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:19:30.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:19:30.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:19:30.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:19:30.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:19:30.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:19:30.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:19:30.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:19:30.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:19:30.433 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:19:30.433 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:19:30.433 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:19:30.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:19:30.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:19:30.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:19:30.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:19:30.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:19:30.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:19:30.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:19:30.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:19:30.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:19:30.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:19:30.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-13T02:19:30.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:19:30.435 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:19:30.435 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T02:19:30.435 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:19:30.436 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T02:19:30.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:19:30.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:19:30.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:19:30.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:19:30.453 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:30.440+0000 7f5134337a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:19:30.454 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:30.440+0000 7f5134337a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:19:30.455 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:30.440+0000 7f5134337a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:19:30.598 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:19:30.598 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:19:30.598 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:19:30.599 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:19:30.599 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:19:30.599 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:19:30.599 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:19:30.599 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:19:30.599 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:19:30.599 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:19:30.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:19:30.906 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:30.892+0000 7f5134337a40 -1 Falling back to public interface 2026-02-13T02:19:31.488 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:31.472+0000 7f5134337a40 -1 osd.0 167 log_to_monitors true 2026-02-13T02:19:31.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:19:31.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:19:31.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:19:31.760 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:19:31.760 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:19:31.760 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:19:31.917 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 171 up_thru 171 down_at 168 last_clean_interval [159,167) [v2:127.0.0.1:6800/2394051377,v1:127.0.0.1:6801/2394051377] [v2:127.0.0.1:6802/2394051377,v1:127.0.0.1:6803/2394051377] exists,up 4f8d5ce6-a345-424d-8fc8-9eb1e4df45a1 2026-02-13T02:19:31.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:19:31.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:19:31.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:19:31.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:19:31.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:19:31.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:19:31.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:19:31.917 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:19:31.918 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:19:31.918 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:19:31.919 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:19:31.919 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:19:31.919 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:19:32.037 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:19:32.037 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:19:32.037 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:19:32.037 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:19:32.037 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:19:32.038 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:19:32.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:19:32.193 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:19:32.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:19:32.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:19:32.194 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:19:32.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=734439407618 2026-02-13T02:19:32.269 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 734439407618 2026-02-13T02:19:32.269 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407618' 2026-02-13T02:19:32.269 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:19:32.269 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:19:32.343 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=708669603847 2026-02-13T02:19:32.343 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 708669603847 2026-02-13T02:19:32.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407618 1-708669603847' 2026-02-13T02:19:32.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:19:32.344 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-734439407618 2026-02-13T02:19:32.344 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:19:32.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:19:32.346 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-734439407618 2026-02-13T02:19:32.346 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:19:32.348 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=734439407618 2026-02-13T02:19:32.348 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 734439407618' 2026-02-13T02:19:32.348 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 734439407618 2026-02-13T02:19:32.349 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:19:32.516 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 682899800073 -lt 734439407618 2026-02-13T02:19:32.516 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:19:33.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:19:33.518 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:19:33.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407619 -lt 734439407618 2026-02-13T02:19:33.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:19:33.684 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-708669603847 2026-02-13T02:19:33.685 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:19:33.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:19:33.687 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-708669603847 2026-02-13T02:19:33.687 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:19:33.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=708669603847 2026-02-13T02:19:33.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 708669603847' 2026-02-13T02:19:33.688 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 708669603847 2026-02-13T02:19:33.689 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:19:33.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 708669603847 -lt 708669603847 2026-02-13T02:19:33.847 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:19:33.848 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:19:33.848 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:19:34.037 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:19:34.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:19:34.039 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:19:34.039 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:19:34.039 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:19:34.039 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:19:34.039 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:19:34.040 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:19:34.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:19:34.198 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:19:34.198 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:19:34.198 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:19:34.397 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:19:34.397 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:19:34.397 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:19:34.397 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6586: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-13T02:19:34.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:19:34.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:19:34.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:19:34.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:19:34.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-13T02:19:34.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:19:34.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:19:34.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:19:34.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:19:34.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:19:34.399 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:19:34.399 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:19:34.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:19:34.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:19:34.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:19:34.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:19:34.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-13T02:19:34.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:19:34.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:19:34.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:19:34.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:19:34.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:19:34.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-13T02:19:34.890 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2026-02-13T02:19:34.891 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2026-02-13T02:19:34.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:19:34.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:19:34.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:19:34.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:19:34.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:19:34.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:19:34.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:19:34.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:19:34.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:19:34.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:19:34.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:19:34.988 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:19:34.988 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:19:34.988 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:19:34.988 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:19:34.988 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:19:34.988 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:19:34.988 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:19:34.988 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:19:34.988 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:19:34.988 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:19:34.988 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:19:34.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:19:34.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:19:34.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:19:34.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:19:34.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:19:34.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:19:34.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:19:34.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:19:34.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:19:34.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-13T02:19:34.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:19:34.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:19:34.992 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:19:34.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:19:34.992 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:19:34.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:19:34.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:19:34.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:19:34.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:19:35.010 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:34.996+0000 7fbc4eadda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:19:35.011 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:34.996+0000 7fbc4eadda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:19:35.013 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:35.000+0000 7fbc4eadda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:19:35.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:19:35.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:19:35.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:19:35.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:19:35.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:19:35.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:19:35.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:19:35.150 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:19:35.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:19:35.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:19:35.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:19:35.414 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:35.400+0000 7fbc4eadda40 -1 Falling back to public interface 2026-02-13T02:19:35.999 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:35.984+0000 7fbc4eadda40 -1 osd.1 172 log_to_monitors true 2026-02-13T02:19:36.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:19:36.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:19:36.315 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:19:36.315 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:19:36.315 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:19:36.315 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:19:36.476 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 176 up_thru 176 down_at 173 last_clean_interval [165,172) [v2:127.0.0.1:6810/1099076570,v1:127.0.0.1:6811/1099076570] [v2:127.0.0.1:6812/1099076570,v1:127.0.0.1:6813/1099076570] exists,up 65f77df4-d31d-4013-880d-185b2abb9aaf 2026-02-13T02:19:36.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:19:36.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:19:36.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:19:36.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:19:36.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:19:36.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:19:36.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:19:36.477 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:19:36.477 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:19:36.478 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:19:36.478 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:19:36.478 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:19:36.478 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:19:36.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:19:36.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:19:36.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:19:36.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:19:36.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:19:36.595 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:19:36.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:19:36.761 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:19:36.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:19:36.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:19:36.762 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:19:36.836 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=734439407624 2026-02-13T02:19:36.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 734439407624 2026-02-13T02:19:36.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407624' 2026-02-13T02:19:36.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:19:36.837 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:19:36.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=755914244098 2026-02-13T02:19:36.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 755914244098 2026-02-13T02:19:36.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407624 1-755914244098' 2026-02-13T02:19:36.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:19:36.912 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-734439407624 2026-02-13T02:19:36.913 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:19:36.914 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:19:36.915 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-734439407624 2026-02-13T02:19:36.915 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:19:36.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=734439407624 2026-02-13T02:19:36.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 734439407624' 2026-02-13T02:19:36.917 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 734439407624 2026-02-13T02:19:36.917 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:19:37.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407621 -lt 734439407624 2026-02-13T02:19:37.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:19:38.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:19:38.079 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:19:38.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407624 -lt 734439407624 2026-02-13T02:19:38.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:19:38.245 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-755914244098 2026-02-13T02:19:38.245 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:19:38.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:19:38.247 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-755914244098 2026-02-13T02:19:38.247 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:19:38.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=755914244098 2026-02-13T02:19:38.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 755914244098' 2026-02-13T02:19:38.249 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 755914244098 2026-02-13T02:19:38.251 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:19:38.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 755914244098 -lt 755914244098 2026-02-13T02:19:38.411 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:19:38.412 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:19:38.412 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:19:38.611 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:19:38.611 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:19:38.612 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:19:38.612 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:19:38.613 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:19:38.613 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:19:38.613 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:19:38.613 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:19:38.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:19:38.778 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:19:38.778 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:19:38.778 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:19:38.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:19:38.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:19:38.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:19:38.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6588: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ18 corrupt-info 2026-02-13T02:19:38.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:19:38.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:19:38.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:19:38.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:19:38.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 corrupt-info 2026-02-13T02:19:38.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:19:38.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:19:38.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:19:38.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:19:38.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:19:38.975 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:19:38.975 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:19:38.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:19:38.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:19:38.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:19:39.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:19:39.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 corrupt-info 2026-02-13T02:19:39.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:19:39.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:19:39.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:19:39.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:19:39.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:19:39.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 corrupt-info 2026-02-13T02:19:39.440 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2026-02-13T02:19:39.440 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2026-02-13T02:19:39.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:19:39.536 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:19:39.536 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:19:39.536 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:19:39.536 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:19:39.536 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:19:39.536 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:19:39.536 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:19:39.536 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:19:39.536 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:19:39.536 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:19:39.536 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:19:39.537 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:19:39.537 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:19:39.537 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:19:39.537 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:19:39.537 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:19:39.537 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:19:39.537 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:19:39.537 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:19:39.537 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:19:39.537 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:19:39.537 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:19:39.537 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:19:39.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:19:39.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:19:39.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:19:39.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:19:39.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:19:39.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:19:39.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:19:39.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-13T02:19:39.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:19:39.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:19:39.539 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:19:39.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-13T02:19:39.540 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:19:39.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:19:39.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:19:39.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:19:39.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:19:39.555 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:39.540+0000 7f307840ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:19:39.556 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:39.544+0000 7f307840ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:19:39.557 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:39.544+0000 7f307840ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:19:39.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:19:39.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:19:39.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:19:39.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:19:39.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:19:39.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:19:39.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:19:39.702 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:19:39.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:19:39.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:19:39.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:19:39.930 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:39.916+0000 7f307840ca40 -1 Falling back to public interface 2026-02-13T02:19:40.575 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:19:40.560+0000 7f307840ca40 -1 osd.1 177 log_to_monitors true 2026-02-13T02:19:40.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:19:40.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:19:40.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:19:40.867 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:19:40.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:19:40.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:19:41.028 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 181 up_thru 181 down_at 178 last_clean_interval [176,177) [v2:127.0.0.1:6810/2042680692,v1:127.0.0.1:6811/2042680692] [v2:127.0.0.1:6812/2042680692,v1:127.0.0.1:6813/2042680692] exists,up 65f77df4-d31d-4013-880d-185b2abb9aaf 2026-02-13T02:19:41.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:19:41.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:19:41.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:19:41.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:19:41.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:19:41.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:19:41.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:19:41.029 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:19:41.030 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:19:41.030 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:19:41.030 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:19:41.030 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:19:41.030 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:19:41.152 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:19:41.152 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:19:41.152 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:19:41.152 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:19:41.152 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:19:41.153 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:19:41.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:19:41.321 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:19:41.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:19:41.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:19:41.322 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:19:41.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=734439407629 2026-02-13T02:19:41.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 734439407629 2026-02-13T02:19:41.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407629' 2026-02-13T02:19:41.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:19:41.399 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:19:41.474 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=777389080578 2026-02-13T02:19:41.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 777389080578 2026-02-13T02:19:41.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407629 1-777389080578' 2026-02-13T02:19:41.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:19:41.475 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-734439407629 2026-02-13T02:19:41.476 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:19:41.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:19:41.478 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-734439407629 2026-02-13T02:19:41.478 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:19:41.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=734439407629 2026-02-13T02:19:41.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 734439407629' 2026-02-13T02:19:41.479 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 734439407629 2026-02-13T02:19:41.480 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:19:41.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407628 -lt 734439407629 2026-02-13T02:19:41.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:19:42.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:19:42.655 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:19:42.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407628 -lt 734439407629 2026-02-13T02:19:42.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:19:43.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:19:43.819 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:19:43.980 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407631 -lt 734439407629 2026-02-13T02:19:43.980 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:19:43.981 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-777389080578 2026-02-13T02:19:43.981 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:19:43.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:19:43.983 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-777389080578 2026-02-13T02:19:43.983 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:19:43.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=777389080578 2026-02-13T02:19:43.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 777389080578' 2026-02-13T02:19:43.985 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 777389080578 2026-02-13T02:19:43.986 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:19:44.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 777389080580 -lt 777389080578 2026-02-13T02:19:44.151 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:19:44.152 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:19:44.152 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:19:44.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:19:44.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:19:44.350 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:19:44.350 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:19:44.350 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:19:44.350 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:19:44.351 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:19:44.351 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:19:44.506 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:19:44.507 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:19:44.507 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:19:44.508 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:19:44.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:19:44.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:19:44.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:19:44.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-13T02:19:44.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ19 2026-02-13T02:19:44.698 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 19 % 2 2026-02-13T02:19:44.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=1 2026-02-13T02:19:44.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-13T02:19:44.700 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6597: TEST_dual_store_replicated_cluster: get_pg csr_pool ROBJ0 2026-02-13T02:19:44.700 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=csr_pool 2026-02-13T02:19:44.700 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=ROBJ0 2026-02-13T02:19:44.701 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map csr_pool ROBJ0 2026-02-13T02:19:44.701 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T02:19:44.859 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6597: TEST_dual_store_replicated_cluster: local pg=3.0 2026-02-13T02:19:44.859 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6599: TEST_dual_store_replicated_cluster: ceph tell 'osd.*' injectargs -- --osd-max-object-size=1048576 2026-02-13T02:19:44.925 INFO:tasks.workunit.client.0.trial112.stdout:osd.0: {} 2026-02-13T02:19:44.926 INFO:tasks.workunit.client.0.trial112.stderr:osd.0: osd_max_object_size = '' 2026-02-13T02:19:44.935 INFO:tasks.workunit.client.0.trial112.stdout:osd.1: {} 2026-02-13T02:19:44.935 INFO:tasks.workunit.client.0.trial112.stderr:osd.1: osd_max_object_size = '' 2026-02-13T02:19:44.944 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6601: TEST_dual_store_replicated_cluster: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2026-02-13T02:19:44.944 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: local pooltype=rep 2026-02-13T02:19:44.944 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: shift 2026-02-13T02:19:44.944 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: local which=data 2026-02-13T02:19:44.944 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: shift 2026-02-13T02:19:44.944 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: local poolname=csr_pool 2026-02-13T02:19:44.944 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: shift 2026-02-13T02:19:44.944 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: local objname=ROBJ11 2026-02-13T02:19:44.944 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: shift 2026-02-13T02:19:44.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: local dir=td/osd-scrub-repair 2026-02-13T02:19:44.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: shift 2026-02-13T02:19:44.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: local shard_id=0 2026-02-13T02:19:44.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2492: inject_eio: shift 2026-02-13T02:19:44.945 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: get_osds csr_pool ROBJ11 2026-02-13T02:19:44.945 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2026-02-13T02:19:44.945 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ11 2026-02-13T02:19:44.945 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ11 2026-02-13T02:19:44.946 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-13T02:19:45.107 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-13T02:19:45.107 INFO:tasks.workunit.client.0.trial112.stderr:0' 2026-02-13T02:19:45.107 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2026-02-13T02:19:45.108 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: initial_osds=('1' '0') 2026-02-13T02:19:45.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local -a initial_osds 2026-02-13T02:19:45.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: local osd_id=1 2026-02-13T02:19:45.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2496: inject_eio: '[' rep '!=' ec ']' 2026-02-13T02:19:45.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2497: inject_eio: shard_id= 2026-02-13T02:19:45.109 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: cat td/osd-scrub-repair/1/type 2026-02-13T02:19:45.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: type=bluestore 2026-02-13T02:19:45.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2026-02-13T02:19:45.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-13T02:19:45.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2026-02-13T02:19:45.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-13T02:19:45.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-13T02:19:45.111 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-13T02:19:45.111 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2026-02-13T02:19:45.112 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-13T02:19:45.112 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-13T02:19:45.112 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:19:45.112 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:19:45.112 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:19:45.112 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-osd.1.asok 2026-02-13T02:19:45.113 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.35207/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2026-02-13T02:19:45.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-13T02:19:45.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: local loop=0 2026-02-13T02:19:45.175 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2503: inject_eio: grep -q Invalid 2026-02-13T02:19:45.175 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: get_asok_path osd.1 2026-02-13T02:19:45.175 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-13T02:19:45.175 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-13T02:19:45.176 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:19:45.176 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:19:45.176 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:19:45.176 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-osd.1.asok 2026-02-13T02:19:45.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: CEPH_ARGS= 2026-02-13T02:19:45.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: ceph --admin-daemon /tmp/ceph-asok.35207/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2026-02-13T02:19:45.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6602: TEST_dual_store_replicated_cluster: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2026-02-13T02:19:45.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: local pooltype=rep 2026-02-13T02:19:45.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: shift 2026-02-13T02:19:45.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: local which=mdata 2026-02-13T02:19:45.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: shift 2026-02-13T02:19:45.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: local poolname=csr_pool 2026-02-13T02:19:45.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: shift 2026-02-13T02:19:45.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: local objname=ROBJ12 2026-02-13T02:19:45.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: shift 2026-02-13T02:19:45.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: local dir=td/osd-scrub-repair 2026-02-13T02:19:45.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: shift 2026-02-13T02:19:45.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: local shard_id=1 2026-02-13T02:19:45.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2492: inject_eio: shift 2026-02-13T02:19:45.237 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: get_osds csr_pool ROBJ12 2026-02-13T02:19:45.237 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2026-02-13T02:19:45.237 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ12 2026-02-13T02:19:45.238 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ12 2026-02-13T02:19:45.238 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-13T02:19:45.403 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-13T02:19:45.404 INFO:tasks.workunit.client.0.trial112.stderr:0' 2026-02-13T02:19:45.404 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2026-02-13T02:19:45.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: initial_osds=('1' '0') 2026-02-13T02:19:45.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local -a initial_osds 2026-02-13T02:19:45.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: local osd_id=0 2026-02-13T02:19:45.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2496: inject_eio: '[' rep '!=' ec ']' 2026-02-13T02:19:45.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2497: inject_eio: shard_id= 2026-02-13T02:19:45.405 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: cat td/osd-scrub-repair/0/type 2026-02-13T02:19:45.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: type=bluestore 2026-02-13T02:19:45.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2026-02-13T02:19:45.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-13T02:19:45.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=0 2026-02-13T02:19:45.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-13T02:19:45.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-13T02:19:45.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-13T02:19:45.407 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.0 2026-02-13T02:19:45.407 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-13T02:19:45.407 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-13T02:19:45.407 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:19:45.407 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:19:45.407 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:19:45.408 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-osd.0.asok 2026-02-13T02:19:45.408 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.35207/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2026-02-13T02:19:45.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-13T02:19:45.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: local loop=0 2026-02-13T02:19:45.469 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2503: inject_eio: grep -q Invalid 2026-02-13T02:19:45.469 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: get_asok_path osd.0 2026-02-13T02:19:45.470 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-13T02:19:45.470 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-13T02:19:45.470 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:19:45.471 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:19:45.471 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:19:45.471 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-osd.0.asok 2026-02-13T02:19:45.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: CEPH_ARGS= 2026-02-13T02:19:45.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: ceph --admin-daemon /tmp/ceph-asok.35207/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2026-02-13T02:19:45.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6603: TEST_dual_store_replicated_cluster: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2026-02-13T02:19:45.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: local pooltype=rep 2026-02-13T02:19:45.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: shift 2026-02-13T02:19:45.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: local which=data 2026-02-13T02:19:45.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: shift 2026-02-13T02:19:45.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: local poolname=csr_pool 2026-02-13T02:19:45.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: shift 2026-02-13T02:19:45.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: local objname=ROBJ13 2026-02-13T02:19:45.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: shift 2026-02-13T02:19:45.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: local dir=td/osd-scrub-repair 2026-02-13T02:19:45.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: shift 2026-02-13T02:19:45.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: local shard_id=0 2026-02-13T02:19:45.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2492: inject_eio: shift 2026-02-13T02:19:45.531 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: get_osds csr_pool ROBJ13 2026-02-13T02:19:45.532 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2026-02-13T02:19:45.532 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ13 2026-02-13T02:19:45.532 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ13 2026-02-13T02:19:45.532 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-13T02:19:45.695 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-13T02:19:45.695 INFO:tasks.workunit.client.0.trial112.stderr:0' 2026-02-13T02:19:45.696 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2026-02-13T02:19:45.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: initial_osds=('1' '0') 2026-02-13T02:19:45.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local -a initial_osds 2026-02-13T02:19:45.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: local osd_id=1 2026-02-13T02:19:45.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2496: inject_eio: '[' rep '!=' ec ']' 2026-02-13T02:19:45.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2497: inject_eio: shard_id= 2026-02-13T02:19:45.697 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: cat td/osd-scrub-repair/1/type 2026-02-13T02:19:45.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: type=bluestore 2026-02-13T02:19:45.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2026-02-13T02:19:45.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-13T02:19:45.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2026-02-13T02:19:45.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-13T02:19:45.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-13T02:19:45.699 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-13T02:19:45.699 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2026-02-13T02:19:45.699 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-13T02:19:45.699 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-13T02:19:45.699 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:19:45.699 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:19:45.699 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:19:45.700 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-osd.1.asok 2026-02-13T02:19:45.700 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.35207/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2026-02-13T02:19:45.760 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-13T02:19:45.760 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: local loop=0 2026-02-13T02:19:45.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2503: inject_eio: grep -q Invalid 2026-02-13T02:19:45.761 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: get_asok_path osd.1 2026-02-13T02:19:45.761 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-13T02:19:45.761 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-13T02:19:45.762 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:19:45.762 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:19:45.762 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:19:45.763 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-osd.1.asok 2026-02-13T02:19:45.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: CEPH_ARGS= 2026-02-13T02:19:45.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: ceph --admin-daemon /tmp/ceph-asok.35207/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2026-02-13T02:19:45.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6606: TEST_dual_store_replicated_cluster: pg_scrub 3.0 2026-02-13T02:19:45.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=3.0 2026-02-13T02:19:45.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 3.0 2026-02-13T02:19:45.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-13T02:19:45.822 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-13T02:19:45.822 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:19:45.823 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:19:45.823 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:19:45.823 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:19:45.823 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:19:46.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-13T02:19:46.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-13T02:19:46.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-13T02:19:46.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-13T02:19:46.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:19:46.135 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:19:46.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:19:46.298 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:19:46.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:19:46.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:19:46.299 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:19:46.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=734439407635 2026-02-13T02:19:46.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 734439407635 2026-02-13T02:19:46.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407635' 2026-02-13T02:19:46.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:19:46.372 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:19:46.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=777389080584 2026-02-13T02:19:46.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 777389080584 2026-02-13T02:19:46.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407635 1-777389080584' 2026-02-13T02:19:46.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:19:46.444 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-734439407635 2026-02-13T02:19:46.445 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:19:46.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:19:46.446 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-734439407635 2026-02-13T02:19:46.447 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:19:46.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=734439407635 2026-02-13T02:19:46.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 734439407635' 2026-02-13T02:19:46.449 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 734439407635 2026-02-13T02:19:46.449 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:19:46.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407633 -lt 734439407635 2026-02-13T02:19:46.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:19:47.613 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:19:47.614 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:19:47.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407636 -lt 734439407635 2026-02-13T02:19:47.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:19:47.781 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-777389080584 2026-02-13T02:19:47.781 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:19:47.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:19:47.784 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-777389080584 2026-02-13T02:19:47.784 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:19:47.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=777389080584 2026-02-13T02:19:47.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 777389080584' 2026-02-13T02:19:47.786 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 777389080584 2026-02-13T02:19:47.786 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:19:47.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 777389080585 -lt 777389080584 2026-02-13T02:19:47.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-13T02:19:47.953 INFO:tasks.workunit.client.0.trial112.stdout:#---------- 3.0 loop 0 2026-02-13T02:19:47.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-13T02:19:47.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-13T02:19:47.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-13T02:19:47.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-13T02:19:47.954 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-13T02:19:47.954 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-13T02:19:48.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-13T02:19:48.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-13T02:19:48.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-13T02:19:48.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-13T02:19:48.028 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: get_last_scrub_stamp 3.0 2026-02-13T02:19:48.028 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:19:48.028 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:19:48.029 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:19:48.029 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:19:48.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: local last_scrub=2026-02-13T02:17:56.280931+0000 2026-02-13T02:19:48.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_scrub: ceph pg scrub 3.0 2026-02-13T02:19:48.334 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 3.0 on osd.1 to scrub 2026-02-13T02:19:48.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_scrub: wait_for_scrub 3.0 2026-02-13T02:17:56.280931+0000 2026-02-13T02:19:48.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-13T02:19:48.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:17:56.280931+0000 2026-02-13T02:19:48.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:19:48.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:19:48.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:19:48.346 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-13T02:19:48.346 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:19:48.346 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:19:48.347 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:19:48.347 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:19:48.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:17:56.280931+0000 '>' 2026-02-13T02:17:56.280931+0000 2026-02-13T02:19:48.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:19:49.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:19:49.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:19:49.504 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-13T02:19:49.504 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:19:49.504 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:19:49.504 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:19:49.504 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:19:49.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:19:48.499629+0000 '>' 2026-02-13T02:17:56.280931+0000 2026-02-13T02:19:49.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:19:49.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6607: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-13T02:19:49.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6608: TEST_dual_store_replicated_cluster: pg_scrub 3.0 2026-02-13T02:19:49.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=3.0 2026-02-13T02:19:49.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 3.0 2026-02-13T02:19:49.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-13T02:19:49.662 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-13T02:19:49.663 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:19:49.663 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:19:49.663 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:19:49.663 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:19:49.663 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:19:49.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-13T02:19:49.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-13T02:19:49.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-13T02:19:49.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-13T02:19:49.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:19:49.962 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:19:50.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:19:50.126 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:19:50.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:19:50.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:19:50.127 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:19:50.202 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=734439407640 2026-02-13T02:19:50.202 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 734439407640 2026-02-13T02:19:50.202 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407640' 2026-02-13T02:19:50.202 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:19:50.202 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:19:50.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=777389080589 2026-02-13T02:19:50.277 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 777389080589 2026-02-13T02:19:50.277 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407640 1-777389080589' 2026-02-13T02:19:50.277 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:19:50.278 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-734439407640 2026-02-13T02:19:50.278 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:19:50.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:19:50.280 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-734439407640 2026-02-13T02:19:50.280 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:19:50.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=734439407640 2026-02-13T02:19:50.282 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 734439407640 2026-02-13T02:19:50.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 734439407640' 2026-02-13T02:19:50.283 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:19:50.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407638 -lt 734439407640 2026-02-13T02:19:50.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:19:51.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:19:51.450 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:19:51.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407641 -lt 734439407640 2026-02-13T02:19:51.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:19:51.617 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-777389080589 2026-02-13T02:19:51.617 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:19:51.618 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:19:51.619 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-777389080589 2026-02-13T02:19:51.619 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:19:51.621 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=777389080589 2026-02-13T02:19:51.621 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 777389080589 2026-02-13T02:19:51.621 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 777389080589' 2026-02-13T02:19:51.621 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:19:51.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 777389080590 -lt 777389080589 2026-02-13T02:19:51.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-13T02:19:51.782 INFO:tasks.workunit.client.0.trial112.stdout:#---------- 3.0 loop 0 2026-02-13T02:19:51.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-13T02:19:51.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-13T02:19:51.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-13T02:19:51.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-13T02:19:51.784 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-13T02:19:51.784 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-13T02:19:51.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean+inconsistent 2026-02-13T02:19:51.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean+inconsistent == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-13T02:19:51.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-13T02:19:51.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-13T02:19:51.857 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: get_last_scrub_stamp 3.0 2026-02-13T02:19:51.857 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:19:51.857 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:19:51.858 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:19:51.858 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:19:52.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: local last_scrub=2026-02-13T02:19:48.499629+0000 2026-02-13T02:19:52.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_scrub: ceph pg scrub 3.0 2026-02-13T02:19:52.166 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 3.0 on osd.1 to scrub 2026-02-13T02:19:52.176 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_scrub: wait_for_scrub 3.0 2026-02-13T02:19:48.499629+0000 2026-02-13T02:19:52.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-13T02:19:52.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:19:48.499629+0000 2026-02-13T02:19:52.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:19:52.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:19:52.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:19:52.178 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-13T02:19:52.178 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:19:52.178 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:19:52.178 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:19:52.178 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:19:52.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:19:48.499629+0000 '>' 2026-02-13T02:19:48.499629+0000 2026-02-13T02:19:52.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:19:53.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:19:53.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:19:53.338 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-13T02:19:53.339 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:19:53.339 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:19:53.339 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:19:53.339 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:19:53.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:19:52.387716+0000 '>' 2026-02-13T02:19:48.499629+0000 2026-02-13T02:19:53.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:19:53.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6609: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-13T02:19:53.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6610: TEST_dual_store_replicated_cluster: rados list-inconsistent-obj 3.0 2026-02-13T02:19:53.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6610: TEST_dual_store_replicated_cluster: jq .inconsistents 2026-02-13T02:19:53.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6610: TEST_dual_store_replicated_cluster: jq 'def walk(f): 2026-02-13T02:19:53.499 INFO:tasks.workunit.client.0.trial112.stderr: . as $in 2026-02-13T02:19:53.499 INFO:tasks.workunit.client.0.trial112.stderr: | if type == "object" then 2026-02-13T02:19:53.500 INFO:tasks.workunit.client.0.trial112.stderr: reduce keys[] as $key 2026-02-13T02:19:53.500 INFO:tasks.workunit.client.0.trial112.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-13T02:19:53.500 INFO:tasks.workunit.client.0.trial112.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-13T02:19:53.500 INFO:tasks.workunit.client.0.trial112.stderr: else f 2026-02-13T02:19:53.500 INFO:tasks.workunit.client.0.trial112.stderr: end; 2026-02-13T02:19:53.500 INFO:tasks.workunit.client.0.trial112.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-13T02:19:53.500 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-13T02:19:53.500 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-13T02:19:53.500 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-13T02:19:53.500 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-13T02:19:53.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6610: TEST_dual_store_replicated_cluster: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-13T02:19:53.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6611: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-13T02:19:53.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6614: TEST_dual_store_replicated_cluster: pg_deep_scrub 3.0 2026-02-13T02:19:53.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1932: pg_deep_scrub: local pgid=3.0 2026-02-13T02:19:53.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: pg_deep_scrub: wait_for_pg_clean 3.0 2026-02-13T02:19:53.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-13T02:19:53.526 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-13T02:19:53.526 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:19:53.527 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:19:53.527 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:19:53.527 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:19:53.527 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:19:53.835 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-13T02:19:53.835 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-13T02:19:53.835 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-13T02:19:53.835 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-13T02:19:53.835 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:19:53.835 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:19:53.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:19:53.993 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:19:53.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:19:53.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:19:53.994 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:19:54.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=734439407645 2026-02-13T02:19:54.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 734439407645 2026-02-13T02:19:54.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407645' 2026-02-13T02:19:54.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:19:54.069 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:19:54.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=777389080594 2026-02-13T02:19:54.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 777389080594 2026-02-13T02:19:54.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407645 1-777389080594' 2026-02-13T02:19:54.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:19:54.143 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-734439407645 2026-02-13T02:19:54.144 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:19:54.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:19:54.146 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-734439407645 2026-02-13T02:19:54.146 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:19:54.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=734439407645 2026-02-13T02:19:54.148 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 734439407645 2026-02-13T02:19:54.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 734439407645' 2026-02-13T02:19:54.148 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:19:54.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407643 -lt 734439407645 2026-02-13T02:19:54.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:19:55.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:19:55.315 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:19:55.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407646 -lt 734439407645 2026-02-13T02:19:55.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:19:55.474 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-777389080594 2026-02-13T02:19:55.474 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:19:55.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:19:55.476 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-777389080594 2026-02-13T02:19:55.477 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:19:55.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=777389080594 2026-02-13T02:19:55.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 777389080594' 2026-02-13T02:19:55.478 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 777389080594 2026-02-13T02:19:55.479 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:19:55.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 777389080595 -lt 777389080594 2026-02-13T02:19:55.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-13T02:19:55.638 INFO:tasks.workunit.client.0.trial112.stdout:#---------- 3.0 loop 0 2026-02-13T02:19:55.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-13T02:19:55.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-13T02:19:55.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-13T02:19:55.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-13T02:19:55.639 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-13T02:19:55.640 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-13T02:19:55.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean+inconsistent 2026-02-13T02:19:55.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean+inconsistent == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-13T02:19:55.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-13T02:19:55.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-13T02:19:55.716 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-13T02:19:55.716 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:19:55.716 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:19:55.716 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:19:55.716 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-13T02:19:55.877 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: local last_scrub=2026-02-13T02:17:56.280931+0000 2026-02-13T02:19:55.877 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: pg_deep_scrub: ceph pg deep-scrub 3.0 2026-02-13T02:19:56.031 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2026-02-13T02:19:56.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: pg_deep_scrub: wait_for_scrub 3.0 2026-02-13T02:17:56.280931+0000 last_deep_scrub_stamp 2026-02-13T02:19:56.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-13T02:19:56.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:17:56.280931+0000 2026-02-13T02:19:56.042 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_deep_scrub_stamp 2026-02-13T02:19:56.042 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:19:56.042 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:19:56.042 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-13T02:19:56.042 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:19:56.042 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:19:56.042 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:19:56.043 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-13T02:19:56.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:17:56.280931+0000 '>' 2026-02-13T02:17:56.280931+0000 2026-02-13T02:19:56.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:19:57.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:19:57.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:19:57.209 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-13T02:19:57.209 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:19:57.209 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:19:57.210 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:19:57.210 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-13T02:19:57.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:19:56.459251+0000 '>' 2026-02-13T02:17:56.280931+0000 2026-02-13T02:19:57.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:19:57.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6615: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-13T02:19:57.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6616: TEST_dual_store_replicated_cluster: rados list-inconsistent-obj 3.0 2026-02-13T02:19:57.372 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6616: TEST_dual_store_replicated_cluster: jq .inconsistents 2026-02-13T02:19:57.372 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6616: TEST_dual_store_replicated_cluster: jq 'def walk(f): 2026-02-13T02:19:57.372 INFO:tasks.workunit.client.0.trial112.stderr: . as $in 2026-02-13T02:19:57.372 INFO:tasks.workunit.client.0.trial112.stderr: | if type == "object" then 2026-02-13T02:19:57.372 INFO:tasks.workunit.client.0.trial112.stderr: reduce keys[] as $key 2026-02-13T02:19:57.373 INFO:tasks.workunit.client.0.trial112.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-13T02:19:57.373 INFO:tasks.workunit.client.0.trial112.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-13T02:19:57.373 INFO:tasks.workunit.client.0.trial112.stderr: else f 2026-02-13T02:19:57.373 INFO:tasks.workunit.client.0.trial112.stderr: end; 2026-02-13T02:19:57.373 INFO:tasks.workunit.client.0.trial112.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-13T02:19:57.373 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-13T02:19:57.373 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-13T02:19:57.373 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-13T02:19:57.373 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-13T02:19:57.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6616: TEST_dual_store_replicated_cluster: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-13T02:19:57.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6617: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-13T02:19:57.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6620: TEST_dual_store_replicated_cluster: pg_scrub 3.0 2026-02-13T02:19:57.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=3.0 2026-02-13T02:19:57.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 3.0 2026-02-13T02:19:57.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-13T02:19:57.405 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-13T02:19:57.405 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:19:57.406 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:19:57.406 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:19:57.406 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:19:57.406 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:19:57.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-13T02:19:57.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-13T02:19:57.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-13T02:19:57.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-13T02:19:57.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:19:57.722 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:19:57.886 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:19:57.886 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:19:57.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:19:57.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:19:57.887 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:19:57.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=734439407650 2026-02-13T02:19:57.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 734439407650 2026-02-13T02:19:57.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407650' 2026-02-13T02:19:57.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:19:57.960 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:19:58.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=777389080599 2026-02-13T02:19:58.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 777389080599 2026-02-13T02:19:58.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407650 1-777389080599' 2026-02-13T02:19:58.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:19:58.036 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-734439407650 2026-02-13T02:19:58.036 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:19:58.037 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:19:58.038 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-734439407650 2026-02-13T02:19:58.038 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:19:58.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=734439407650 2026-02-13T02:19:58.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 734439407650' 2026-02-13T02:19:58.041 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 734439407650 2026-02-13T02:19:58.041 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:19:58.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407648 -lt 734439407650 2026-02-13T02:19:58.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:19:59.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:19:59.206 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:19:59.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407651 -lt 734439407650 2026-02-13T02:19:59.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:19:59.371 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-777389080599 2026-02-13T02:19:59.371 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:19:59.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:19:59.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-777389080599 2026-02-13T02:19:59.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:19:59.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=777389080599 2026-02-13T02:19:59.376 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 777389080599 2026-02-13T02:19:59.376 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 777389080599' 2026-02-13T02:19:59.376 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:19:59.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 777389080600 -lt 777389080599 2026-02-13T02:19:59.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-13T02:19:59.550 INFO:tasks.workunit.client.0.trial112.stdout:#---------- 3.0 loop 0 2026-02-13T02:19:59.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-13T02:19:59.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-13T02:19:59.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-13T02:19:59.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-13T02:19:59.551 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-13T02:19:59.551 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-13T02:19:59.626 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean+inconsistent 2026-02-13T02:19:59.626 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean+inconsistent == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-13T02:19:59.626 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-13T02:19:59.626 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-13T02:19:59.627 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: get_last_scrub_stamp 3.0 2026-02-13T02:19:59.627 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:19:59.627 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:19:59.627 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:19:59.628 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:19:59.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: local last_scrub=2026-02-13T02:19:56.459251+0000 2026-02-13T02:19:59.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_scrub: ceph pg scrub 3.0 2026-02-13T02:19:59.945 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 3.0 on osd.1 to scrub 2026-02-13T02:19:59.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_scrub: wait_for_scrub 3.0 2026-02-13T02:19:56.459251+0000 2026-02-13T02:19:59.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-13T02:19:59.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:19:56.459251+0000 2026-02-13T02:19:59.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:19:59.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:19:59.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:19:59.956 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-13T02:19:59.956 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:19:59.956 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:19:59.957 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:19:59.957 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:20:00.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:19:56.459251+0000 '>' 2026-02-13T02:19:56.459251+0000 2026-02-13T02:20:00.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:20:01.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:20:01.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:20:01.121 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-13T02:20:01.121 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:20:01.121 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:20:01.122 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:20:01.122 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:20:01.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:20:00.557457+0000 '>' 2026-02-13T02:19:56.459251+0000 2026-02-13T02:20:01.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:20:01.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6621: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-13T02:20:01.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6622: TEST_dual_store_replicated_cluster: rados list-inconsistent-obj 3.0 2026-02-13T02:20:01.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6622: TEST_dual_store_replicated_cluster: jq .inconsistents 2026-02-13T02:20:01.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6622: TEST_dual_store_replicated_cluster: jq 'def walk(f): 2026-02-13T02:20:01.279 INFO:tasks.workunit.client.0.trial112.stderr: . as $in 2026-02-13T02:20:01.279 INFO:tasks.workunit.client.0.trial112.stderr: | if type == "object" then 2026-02-13T02:20:01.279 INFO:tasks.workunit.client.0.trial112.stderr: reduce keys[] as $key 2026-02-13T02:20:01.279 INFO:tasks.workunit.client.0.trial112.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-13T02:20:01.280 INFO:tasks.workunit.client.0.trial112.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-13T02:20:01.280 INFO:tasks.workunit.client.0.trial112.stderr: else f 2026-02-13T02:20:01.280 INFO:tasks.workunit.client.0.trial112.stderr: end; 2026-02-13T02:20:01.280 INFO:tasks.workunit.client.0.trial112.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-13T02:20:01.280 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-13T02:20:01.280 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-13T02:20:01.280 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-13T02:20:01.280 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-13T02:20:01.280 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6622: TEST_dual_store_replicated_cluster: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-13T02:20:01.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6623: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-13T02:20:01.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6626: TEST_dual_store_replicated_cluster: diff -u td/osd-scrub-repair/dp_results.json td/osd-scrub-repair/sh2_results.json 2026-02-13T02:20:01.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6630: TEST_dual_store_replicated_cluster: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2026-02-13T02:20:01.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: local pooltype=rep 2026-02-13T02:20:01.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: shift 2026-02-13T02:20:01.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: local which=mdata 2026-02-13T02:20:01.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: shift 2026-02-13T02:20:01.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: local poolname=csr_pool 2026-02-13T02:20:01.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: shift 2026-02-13T02:20:01.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: local objname=ROBJ13 2026-02-13T02:20:01.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: shift 2026-02-13T02:20:01.315 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: local dir=td/osd-scrub-repair 2026-02-13T02:20:01.315 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: shift 2026-02-13T02:20:01.315 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: local shard_id=1 2026-02-13T02:20:01.315 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2492: inject_eio: shift 2026-02-13T02:20:01.315 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: get_osds csr_pool ROBJ13 2026-02-13T02:20:01.315 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2026-02-13T02:20:01.315 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ13 2026-02-13T02:20:01.315 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ13 2026-02-13T02:20:01.316 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-13T02:20:01.476 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-13T02:20:01.476 INFO:tasks.workunit.client.0.trial112.stderr:0' 2026-02-13T02:20:01.476 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2026-02-13T02:20:01.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: initial_osds=('1' '0') 2026-02-13T02:20:01.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local -a initial_osds 2026-02-13T02:20:01.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: local osd_id=0 2026-02-13T02:20:01.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2496: inject_eio: '[' rep '!=' ec ']' 2026-02-13T02:20:01.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2497: inject_eio: shard_id= 2026-02-13T02:20:01.477 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: cat td/osd-scrub-repair/0/type 2026-02-13T02:20:01.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: type=bluestore 2026-02-13T02:20:01.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2026-02-13T02:20:01.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-13T02:20:01.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=0 2026-02-13T02:20:01.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-13T02:20:01.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-13T02:20:01.479 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-13T02:20:01.480 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.0 2026-02-13T02:20:01.480 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-13T02:20:01.480 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-13T02:20:01.480 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:20:01.480 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:20:01.480 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:20:01.480 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-osd.0.asok 2026-02-13T02:20:01.481 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.35207/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2026-02-13T02:20:01.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-13T02:20:01.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: local loop=0 2026-02-13T02:20:01.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2503: inject_eio: grep -q Invalid 2026-02-13T02:20:01.543 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: get_asok_path osd.0 2026-02-13T02:20:01.543 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-13T02:20:01.543 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-13T02:20:01.544 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:20:01.544 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:20:01.544 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:20:01.544 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-osd.0.asok 2026-02-13T02:20:01.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: CEPH_ARGS= 2026-02-13T02:20:01.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: ceph --admin-daemon /tmp/ceph-asok.35207/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2026-02-13T02:20:01.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6632: TEST_dual_store_replicated_cluster: pg_deep_scrub 3.0 2026-02-13T02:20:01.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1932: pg_deep_scrub: local pgid=3.0 2026-02-13T02:20:01.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: pg_deep_scrub: wait_for_pg_clean 3.0 2026-02-13T02:20:01.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-13T02:20:01.604 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-13T02:20:01.605 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:20:01.605 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:20:01.605 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:20:01.605 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:20:01.605 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:20:01.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-13T02:20:01.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-13T02:20:01.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-13T02:20:01.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-13T02:20:01.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:20:01.923 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:20:02.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:20:02.085 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:20:02.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:20:02.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:20:02.086 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:20:02.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=734439407655 2026-02-13T02:20:02.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 734439407655 2026-02-13T02:20:02.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407655' 2026-02-13T02:20:02.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:20:02.162 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:20:02.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=777389080604 2026-02-13T02:20:02.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 777389080604 2026-02-13T02:20:02.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407655 1-777389080604' 2026-02-13T02:20:02.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:20:02.236 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-734439407655 2026-02-13T02:20:02.236 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:20:02.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:20:02.238 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-734439407655 2026-02-13T02:20:02.238 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:20:02.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=734439407655 2026-02-13T02:20:02.240 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 734439407655 2026-02-13T02:20:02.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 734439407655' 2026-02-13T02:20:02.241 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:20:02.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407653 -lt 734439407655 2026-02-13T02:20:02.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:20:03.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:20:03.400 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:20:03.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407656 -lt 734439407655 2026-02-13T02:20:03.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:20:03.566 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-777389080604 2026-02-13T02:20:03.566 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:20:03.568 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:20:03.568 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-777389080604 2026-02-13T02:20:03.569 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:20:03.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=777389080604 2026-02-13T02:20:03.571 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 777389080604 2026-02-13T02:20:03.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 777389080604' 2026-02-13T02:20:03.571 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:20:03.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 777389080605 -lt 777389080604 2026-02-13T02:20:03.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-13T02:20:03.743 INFO:tasks.workunit.client.0.trial112.stdout:#---------- 3.0 loop 0 2026-02-13T02:20:03.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-13T02:20:03.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-13T02:20:03.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-13T02:20:03.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-13T02:20:03.744 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-13T02:20:03.744 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-13T02:20:03.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean+inconsistent 2026-02-13T02:20:03.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean+inconsistent == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-13T02:20:03.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-13T02:20:03.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-13T02:20:03.818 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-13T02:20:03.818 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:20:03.819 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:20:03.819 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:20:03.819 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-13T02:20:03.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: local last_scrub=2026-02-13T02:19:56.459251+0000 2026-02-13T02:20:03.980 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: pg_deep_scrub: ceph pg deep-scrub 3.0 2026-02-13T02:20:04.134 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2026-02-13T02:20:04.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: pg_deep_scrub: wait_for_scrub 3.0 2026-02-13T02:19:56.459251+0000 last_deep_scrub_stamp 2026-02-13T02:20:04.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-13T02:20:04.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:19:56.459251+0000 2026-02-13T02:20:04.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_deep_scrub_stamp 2026-02-13T02:20:04.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:20:04.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:20:04.146 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-13T02:20:04.147 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:20:04.147 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:20:04.147 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:20:04.147 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-13T02:20:04.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:19:56.459251+0000 '>' 2026-02-13T02:19:56.459251+0000 2026-02-13T02:20:04.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:20:05.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:20:05.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:20:05.306 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-13T02:20:05.306 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:20:05.307 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:20:05.307 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:20:05.307 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-13T02:20:05.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:20:04.626294+0000 '>' 2026-02-13T02:19:56.459251+0000 2026-02-13T02:20:05.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:20:05.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6634: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-13T02:20:05.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6635: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-13T02:20:05.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6637: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-13T02:20:05.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6641: TEST_dual_store_replicated_cluster: rados list-inconsistent-obj 3.0 2026-02-13T02:20:05.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6641: TEST_dual_store_replicated_cluster: jq .inconsistents 2026-02-13T02:20:05.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6641: TEST_dual_store_replicated_cluster: jq 'def walk(f): 2026-02-13T02:20:05.468 INFO:tasks.workunit.client.0.trial112.stderr: . as $in 2026-02-13T02:20:05.468 INFO:tasks.workunit.client.0.trial112.stderr: | if type == "object" then 2026-02-13T02:20:05.468 INFO:tasks.workunit.client.0.trial112.stderr: reduce keys[] as $key 2026-02-13T02:20:05.468 INFO:tasks.workunit.client.0.trial112.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-13T02:20:05.468 INFO:tasks.workunit.client.0.trial112.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-13T02:20:05.468 INFO:tasks.workunit.client.0.trial112.stderr: else f 2026-02-13T02:20:05.468 INFO:tasks.workunit.client.0.trial112.stderr: end; 2026-02-13T02:20:05.468 INFO:tasks.workunit.client.0.trial112.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-13T02:20:05.468 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-13T02:20:05.468 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-13T02:20:05.468 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-13T02:20:05.468 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-13T02:20:05.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6642: TEST_dual_store_replicated_cluster: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-13T02:20:05.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6644: TEST_dual_store_replicated_cluster: rados list-inconsistent-obj 3.0 2026-02-13T02:20:05.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6644: TEST_dual_store_replicated_cluster: jq 'def walk(f): 2026-02-13T02:20:05.500 INFO:tasks.workunit.client.0.trial112.stderr: . as $in 2026-02-13T02:20:05.500 INFO:tasks.workunit.client.0.trial112.stderr: | if type == "object" then 2026-02-13T02:20:05.500 INFO:tasks.workunit.client.0.trial112.stderr: reduce keys[] as $key 2026-02-13T02:20:05.500 INFO:tasks.workunit.client.0.trial112.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-13T02:20:05.500 INFO:tasks.workunit.client.0.trial112.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-13T02:20:05.500 INFO:tasks.workunit.client.0.trial112.stderr: else f 2026-02-13T02:20:05.501 INFO:tasks.workunit.client.0.trial112.stderr: end; 2026-02-13T02:20:05.501 INFO:tasks.workunit.client.0.trial112.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-13T02:20:05.501 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-13T02:20:05.501 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-13T02:20:05.501 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-13T02:20:05.501 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-13T02:20:05.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6644: TEST_dual_store_replicated_cluster: jq 'del(.inconsistents[] | select(.object.name == "ROBJ13"))' 2026-02-13T02:20:05.502 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6645: TEST_dual_store_replicated_cluster: jq .inconsistents 2026-02-13T02:20:05.502 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6645: TEST_dual_store_replicated_cluster: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-13T02:20:05.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6646: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-13T02:20:05.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6649: TEST_dual_store_replicated_cluster: pg_scrub 3.0 2026-02-13T02:20:05.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=3.0 2026-02-13T02:20:05.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 3.0 2026-02-13T02:20:05.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-13T02:20:05.541 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-13T02:20:05.542 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:20:05.542 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:20:05.542 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:20:05.542 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:20:05.542 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:20:05.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-13T02:20:05.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-13T02:20:05.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-13T02:20:05.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-13T02:20:05.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:20:05.851 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:20:06.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:20:06.009 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:20:06.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:20:06.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:20:06.010 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:20:06.086 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=734439407660 2026-02-13T02:20:06.086 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 734439407660 2026-02-13T02:20:06.086 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407660' 2026-02-13T02:20:06.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:20:06.087 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:20:06.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=777389080609 2026-02-13T02:20:06.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 777389080609 2026-02-13T02:20:06.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407660 1-777389080609' 2026-02-13T02:20:06.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:20:06.164 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-734439407660 2026-02-13T02:20:06.164 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:20:06.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:20:06.166 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-734439407660 2026-02-13T02:20:06.167 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:20:06.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=734439407660 2026-02-13T02:20:06.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 734439407660' 2026-02-13T02:20:06.169 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 734439407660 2026-02-13T02:20:06.169 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:20:06.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407658 -lt 734439407660 2026-02-13T02:20:06.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:20:07.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:20:07.329 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:20:07.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407661 -lt 734439407660 2026-02-13T02:20:07.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:20:07.494 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-777389080609 2026-02-13T02:20:07.495 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:20:07.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:20:07.497 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-777389080609 2026-02-13T02:20:07.497 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:20:07.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=777389080609 2026-02-13T02:20:07.499 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 777389080609 2026-02-13T02:20:07.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 777389080609' 2026-02-13T02:20:07.500 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:20:07.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 777389080610 -lt 777389080609 2026-02-13T02:20:07.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-13T02:20:07.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-13T02:20:07.669 INFO:tasks.workunit.client.0.trial112.stdout:#---------- 3.0 loop 0 2026-02-13T02:20:07.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-13T02:20:07.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-13T02:20:07.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-13T02:20:07.670 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-13T02:20:07.670 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-13T02:20:07.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean+inconsistent 2026-02-13T02:20:07.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean+inconsistent == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-13T02:20:07.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-13T02:20:07.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-13T02:20:07.745 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: get_last_scrub_stamp 3.0 2026-02-13T02:20:07.745 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:20:07.745 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:20:07.745 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:20:07.746 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:20:07.901 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: local last_scrub=2026-02-13T02:20:04.626294+0000 2026-02-13T02:20:07.901 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_scrub: ceph pg scrub 3.0 2026-02-13T02:20:08.053 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 3.0 on osd.1 to scrub 2026-02-13T02:20:08.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_scrub: wait_for_scrub 3.0 2026-02-13T02:20:04.626294+0000 2026-02-13T02:20:08.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-13T02:20:08.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:20:04.626294+0000 2026-02-13T02:20:08.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:20:08.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:20:08.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:20:08.064 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-13T02:20:08.064 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:20:08.064 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:20:08.064 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:20:08.064 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:20:08.222 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:20:04.626294+0000 '>' 2026-02-13T02:20:04.626294+0000 2026-02-13T02:20:08.222 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:20:09.223 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:20:09.223 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:20:09.224 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-13T02:20:09.224 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-13T02:20:09.224 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:20:09.225 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:20:09.225 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-13T02:20:09.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:20:08.595043+0000 '>' 2026-02-13T02:20:04.626294+0000 2026-02-13T02:20:09.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:20:09.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6651: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-13T02:20:09.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6652: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-13T02:20:09.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6654: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-13T02:20:09.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6658: TEST_dual_store_replicated_cluster: rados list-inconsistent-obj 3.0 2026-02-13T02:20:09.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6658: TEST_dual_store_replicated_cluster: jq .inconsistents 2026-02-13T02:20:09.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6658: TEST_dual_store_replicated_cluster: jq 'def walk(f): 2026-02-13T02:20:09.380 INFO:tasks.workunit.client.0.trial112.stderr: . as $in 2026-02-13T02:20:09.380 INFO:tasks.workunit.client.0.trial112.stderr: | if type == "object" then 2026-02-13T02:20:09.381 INFO:tasks.workunit.client.0.trial112.stderr: reduce keys[] as $key 2026-02-13T02:20:09.381 INFO:tasks.workunit.client.0.trial112.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-13T02:20:09.381 INFO:tasks.workunit.client.0.trial112.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-13T02:20:09.381 INFO:tasks.workunit.client.0.trial112.stderr: else f 2026-02-13T02:20:09.381 INFO:tasks.workunit.client.0.trial112.stderr: end; 2026-02-13T02:20:09.381 INFO:tasks.workunit.client.0.trial112.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-13T02:20:09.381 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-13T02:20:09.381 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-13T02:20:09.381 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-13T02:20:09.381 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-13T02:20:09.381 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6658: TEST_dual_store_replicated_cluster: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-13T02:20:09.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6660: TEST_dual_store_replicated_cluster: rados list-inconsistent-obj 3.0 2026-02-13T02:20:09.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6660: TEST_dual_store_replicated_cluster: jq 'del(.inconsistents[] | select(.object.name == "ROBJ13"))' 2026-02-13T02:20:09.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6660: TEST_dual_store_replicated_cluster: jq 'def walk(f): 2026-02-13T02:20:09.413 INFO:tasks.workunit.client.0.trial112.stderr: . as $in 2026-02-13T02:20:09.413 INFO:tasks.workunit.client.0.trial112.stderr: | if type == "object" then 2026-02-13T02:20:09.413 INFO:tasks.workunit.client.0.trial112.stderr: reduce keys[] as $key 2026-02-13T02:20:09.413 INFO:tasks.workunit.client.0.trial112.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-13T02:20:09.413 INFO:tasks.workunit.client.0.trial112.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-13T02:20:09.414 INFO:tasks.workunit.client.0.trial112.stderr: else f 2026-02-13T02:20:09.414 INFO:tasks.workunit.client.0.trial112.stderr: end; 2026-02-13T02:20:09.414 INFO:tasks.workunit.client.0.trial112.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-13T02:20:09.414 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-13T02:20:09.414 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-13T02:20:09.414 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-13T02:20:09.414 INFO:tasks.workunit.client.0.trial112.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-13T02:20:09.414 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6661: TEST_dual_store_replicated_cluster: jq .inconsistents 2026-02-13T02:20:09.414 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6661: TEST_dual_store_replicated_cluster: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-13T02:20:09.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6665: TEST_dual_store_replicated_cluster: diff -u td/osd-scrub-repair/dpPart2_w13_results.json td/osd-scrub-repair/sh2Part2_w13_results.json 2026-02-13T02:20:09.453 INFO:tasks.workunit.client.0.trial112.stdout:--- td/osd-scrub-repair/dpPart2_w13_results.json 2026-02-13 02:20:05.484246709 +0000 2026-02-13T02:20:09.453 INFO:tasks.workunit.client.0.trial112.stdout:+++ td/osd-scrub-repair/sh2Part2_w13_results.json 2026-02-13 02:20:09.396188739 +0000 2026-02-13T02:20:09.453 INFO:tasks.workunit.client.0.trial112.stdout:@@ -233,6 +233,38 @@ 2026-02-13T02:20:09.453 INFO:tasks.workunit.client.0.trial112.stdout: "nspace": "", 2026-02-13T02:20:09.453 INFO:tasks.workunit.client.0.trial112.stdout: "snap": "head" 2026-02-13T02:20:09.454 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:20:09.454 INFO:tasks.workunit.client.0.trial112.stdout:+ "selected_object_info": { 2026-02-13T02:20:09.454 INFO:tasks.workunit.client.0.trial112.stdout:+ "alloc_hint_flags": 0, 2026-02-13T02:20:09.454 INFO:tasks.workunit.client.0.trial112.stdout:+ "data_digest": "0x2ddbf8f5", 2026-02-13T02:20:09.454 INFO:tasks.workunit.client.0.trial112.stdout:+ "expected_object_size": 0, 2026-02-13T02:20:09.454 INFO:tasks.workunit.client.0.trial112.stdout:+ "expected_write_size": 0, 2026-02-13T02:20:09.454 INFO:tasks.workunit.client.0.trial112.stdout:+ "flags": [ 2026-02-13T02:20:09.454 INFO:tasks.workunit.client.0.trial112.stdout:+ "dirty", 2026-02-13T02:20:09.454 INFO:tasks.workunit.client.0.trial112.stdout:+ "omap", 2026-02-13T02:20:09.454 INFO:tasks.workunit.client.0.trial112.stdout:+ "data_digest", 2026-02-13T02:20:09.454 INFO:tasks.workunit.client.0.trial112.stdout:+ "omap_digest" 2026-02-13T02:20:09.454 INFO:tasks.workunit.client.0.trial112.stdout:+ ], 2026-02-13T02:20:09.455 INFO:tasks.workunit.client.0.trial112.stdout:+ "lost": 0, 2026-02-13T02:20:09.455 INFO:tasks.workunit.client.0.trial112.stdout:+ "manifest": { 2026-02-13T02:20:09.455 INFO:tasks.workunit.client.0.trial112.stdout:+ "type": 0 2026-02-13T02:20:09.455 INFO:tasks.workunit.client.0.trial112.stdout:+ }, 2026-02-13T02:20:09.455 INFO:tasks.workunit.client.0.trial112.stdout:+ "oid": { 2026-02-13T02:20:09.455 INFO:tasks.workunit.client.0.trial112.stdout:+ "hash": 2682806379, 2026-02-13T02:20:09.455 INFO:tasks.workunit.client.0.trial112.stdout:+ "key": "", 2026-02-13T02:20:09.455 INFO:tasks.workunit.client.0.trial112.stdout:+ "max": 0, 2026-02-13T02:20:09.455 INFO:tasks.workunit.client.0.trial112.stdout:+ "namespace": "", 2026-02-13T02:20:09.455 INFO:tasks.workunit.client.0.trial112.stdout:+ "oid": "ROBJ13", 2026-02-13T02:20:09.455 INFO:tasks.workunit.client.0.trial112.stdout:+ "pool": 3, 2026-02-13T02:20:09.455 INFO:tasks.workunit.client.0.trial112.stdout:+ "snapid": -2 2026-02-13T02:20:09.455 INFO:tasks.workunit.client.0.trial112.stdout:+ }, 2026-02-13T02:20:09.456 INFO:tasks.workunit.client.0.trial112.stdout:+ "omap_digest": "0x6441854d", 2026-02-13T02:20:09.456 INFO:tasks.workunit.client.0.trial112.stdout:+ "shard_versions": [], 2026-02-13T02:20:09.456 INFO:tasks.workunit.client.0.trial112.stdout:+ "size": 7, 2026-02-13T02:20:09.456 INFO:tasks.workunit.client.0.trial112.stdout:+ "truncate_seq": 0, 2026-02-13T02:20:09.456 INFO:tasks.workunit.client.0.trial112.stdout:+ "truncate_size": 0, 2026-02-13T02:20:09.456 INFO:tasks.workunit.client.0.trial112.stdout:+ "user_version": 39, 2026-02-13T02:20:09.456 INFO:tasks.workunit.client.0.trial112.stdout:+ "watchers": {} 2026-02-13T02:20:09.456 INFO:tasks.workunit.client.0.trial112.stdout:+ }, 2026-02-13T02:20:09.456 INFO:tasks.workunit.client.0.trial112.stdout: "shards": [ 2026-02-13T02:20:09.456 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:20:09.456 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [ 2026-02-13T02:20:09.456 INFO:tasks.workunit.client.0.trial112.stdout:@@ -242,17 +274,14 @@ 2026-02-13T02:20:09.456 INFO:tasks.workunit.client.0.trial112.stdout: "primary": false 2026-02-13T02:20:09.457 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:20:09.457 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:20:09.457 INFO:tasks.workunit.client.0.trial112.stdout:- "errors": [ 2026-02-13T02:20:09.457 INFO:tasks.workunit.client.0.trial112.stdout:- "read_error" 2026-02-13T02:20:09.457 INFO:tasks.workunit.client.0.trial112.stdout:- ], 2026-02-13T02:20:09.457 INFO:tasks.workunit.client.0.trial112.stdout:+ "errors": [], 2026-02-13T02:20:09.457 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 1, 2026-02-13T02:20:09.457 INFO:tasks.workunit.client.0.trial112.stdout: "primary": true, 2026-02-13T02:20:09.457 INFO:tasks.workunit.client.0.trial112.stdout: "size": 7 2026-02-13T02:20:09.457 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:20:09.457 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:20:09.457 INFO:tasks.workunit.client.0.trial112.stdout: "union_shard_errors": [ 2026-02-13T02:20:09.458 INFO:tasks.workunit.client.0.trial112.stdout:- "stat_error", 2026-02-13T02:20:09.458 INFO:tasks.workunit.client.0.trial112.stdout:- "read_error" 2026-02-13T02:20:09.458 INFO:tasks.workunit.client.0.trial112.stdout:+ "stat_error" 2026-02-13T02:20:09.458 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:20:09.458 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:20:09.458 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:20:09.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6666: TEST_dual_store_replicated_cluster: diff -u td/osd-scrub-repair/dpPart2_wo13_results.json td/osd-scrub-repair/shPart2_wo13_results.json 2026-02-13T02:20:09.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6668: TEST_dual_store_replicated_cluster: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2026-02-13T02:20:09.662 INFO:tasks.workunit.client.0.trial112.stderr:pool 'csr_pool' removed 2026-02-13T02:20:09.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6669: TEST_dual_store_replicated_cluster: return 0 2026-02-13T02:20:09.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-13T02:20:09.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:20:09.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:20:09.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:20:09.673 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:20:09.673 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:20:09.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:20:09.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:20:09.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:20:09.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:20:09.786 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:20:09.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:20:09.787 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:20:09.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:20:09.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:20:09.790 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:20:09.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:20:09.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:20:09.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:20:09.792 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:20:09.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:20:09.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:20:09.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:20:09.827 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:20:09.827 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:20:09.827 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:20:09.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:20:09.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:20:09.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:20:09.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-13T02:20:09.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-13T02:20:09.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-13T02:20:09.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-13T02:20:09.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:20:09.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:20:09.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:20:09.831 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:20:09.831 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:20:09.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:20:09.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:20:09.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:20:09.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:20:09.833 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:20:09.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:20:09.834 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:20:09.836 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:20:09.836 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:20:09.836 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:20:09.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:20:09.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:20:09.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:20:09.838 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:20:09.839 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:20:09.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:20:09.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:20:09.842 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:20:09.842 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:20:09.842 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:20:09.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:20:09.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:20:09.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:20:09.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-13T02:20:09.845 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T02:20:09.846 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:20:09.846 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:20:09.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.35207 2026-02-13T02:20:09.848 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T02:20:09.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-13T02:20:09.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T02:20:09.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-13T02:20:09.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_list_missing_erasure_coded_appends td/osd-scrub-repair 2026-02-13T02:20:09.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1037: TEST_list_missing_erasure_coded_appends: list_missing_erasure_coded td/osd-scrub-repair false 2026-02-13T02:20:09.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:975: list_missing_erasure_coded: local dir=td/osd-scrub-repair 2026-02-13T02:20:09.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:976: list_missing_erasure_coded: local allow_overwrites=false 2026-02-13T02:20:09.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:977: list_missing_erasure_coded: local poolname=ecpool 2026-02-13T02:20:09.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:979: list_missing_erasure_coded: run_mon td/osd-scrub-repair a 2026-02-13T02:20:09.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-13T02:20:09.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-13T02:20:09.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-13T02:20:09.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-13T02:20:09.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-13T02:20:09.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-13T02:20:09.872 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-13T02:20:09.872 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:20:09.872 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:20:09.873 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:20:09.873 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:20:09.873 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:20:09.873 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:20:09.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-13T02:20:09.905 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-13T02:20:09.906 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-13T02:20:09.906 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:20:09.906 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:20:09.906 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-13T02:20:09.907 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-13T02:20:09.907 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:20:09.907 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:20:09.907 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:20:09.908 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:20:09.908 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:20:09.908 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:20:09.909 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:20:09.909 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:20:09.909 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get fsid 2026-02-13T02:20:09.966 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-13T02:20:09.966 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:20:09.966 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:20:09.966 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-13T02:20:09.967 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-13T02:20:09.967 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:20:09.967 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:20:09.967 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:20:09.968 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:20:09.968 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:20:09.969 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:20:09.969 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:20:09.969 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:20:09.969 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get mon_host 2026-02-13T02:20:10.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:980: list_missing_erasure_coded: run_mgr td/osd-scrub-repair x 2026-02-13T02:20:10.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-13T02:20:10.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-13T02:20:10.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-13T02:20:10.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-13T02:20:10.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-13T02:20:10.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-13T02:20:10.120 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-13T02:20:10.121 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:20:10.121 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:20:10.121 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:20:10.121 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:20:10.122 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:20:10.122 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:20:10.122 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:20:10.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:20:10.143 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:981: list_missing_erasure_coded: seq 0 2 2026-02-13T02:20:10.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:981: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-13T02:20:10.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:982: list_missing_erasure_coded: run_osd td/osd-scrub-repair 0 2026-02-13T02:20:10.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:20:10.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:20:10.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-13T02:20:10.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:20:10.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:20:10.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:20:10.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:20:10.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:20:10.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:20:10.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:20:10.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:20:10.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:20:10.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:20:10.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:20:10.147 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:20:10.147 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:20:10.147 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:20:10.147 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:20:10.147 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:20:10.147 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:20:10.147 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:20:10.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:20:10.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:20:10.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:20:10.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:20:10.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:20:10.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:20:10.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:20:10.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:20:10.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:20:10.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:20:10.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:20:10.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:20:10.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:20:10.150 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:20:10.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=25c08f3b-b1f6-4e4d-b941-e57ea07c391e 2026-02-13T02:20:10.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 25c08f3b-b1f6-4e4d-b941-e57ea07c391e' 2026-02-13T02:20:10.150 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 25c08f3b-b1f6-4e4d-b941-e57ea07c391e 2026-02-13T02:20:10.151 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:20:10.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBaio5polv9CBAAJlv4fyZX4Hk9GpkRVZWzrQ== 2026-02-13T02:20:10.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBaio5polv9CBAAJlv4fyZX4Hk9GpkRVZWzrQ=="}' 2026-02-13T02:20:10.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 25c08f3b-b1f6-4e4d-b941-e57ea07c391e -i td/osd-scrub-repair/0/new.json 2026-02-13T02:20:10.251 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:20:10.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-13T02:20:10.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBaio5polv9CBAAJlv4fyZX4Hk9GpkRVZWzrQ== --osd-uuid 25c08f3b-b1f6-4e4d-b941-e57ea07c391e 2026-02-13T02:20:10.276 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:10.264+0000 7ff49703ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:10.278 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:10.264+0000 7ff49703ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:10.279 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:10.264+0000 7ff49703ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:10.279 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:10.264+0000 7ff49703ea40 -1 bdev(0x5587a4000c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-13T02:20:10.280 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:10.264+0000 7ff49703ea40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-13T02:20:11.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-13T02:20:11.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:20:11.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-13T02:20:11.077 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T02:20:11.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:20:11.185 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T02:20:11.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-13T02:20:11.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:20:11.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:20:11.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:20:11.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:20:11.208 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:11.192+0000 7f9997ff1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:11.210 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:11.196+0000 7f9997ff1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:11.212 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:11.196+0000 7f9997ff1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:11.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-13T02:20:11.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:20:11.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:20:11.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:20:11.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:20:11.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:20:11.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:20:11.283 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:20:11.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:20:11.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:20:11.376 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:20:11.602 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:11.588+0000 7f9997ff1a40 -1 Falling back to public interface 2026-02-13T02:20:12.349 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:12.336+0000 7f9997ff1a40 -1 osd.0 0 log_to_monitors true 2026-02-13T02:20:12.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:20:12.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:20:12.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:20:12.378 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:20:12.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:20:12.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:20:12.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:20:13.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:20:13.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:20:13.473 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:20:13.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:20:13.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:20:13.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:20:13.638 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2374259400,v1:127.0.0.1:6801/2374259400] [v2:127.0.0.1:6802/2374259400,v1:127.0.0.1:6803/2374259400] exists,up 25c08f3b-b1f6-4e4d-b941-e57ea07c391e 2026-02-13T02:20:13.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:20:13.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:20:13.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:20:13.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:981: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-13T02:20:13.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:982: list_missing_erasure_coded: run_osd td/osd-scrub-repair 1 2026-02-13T02:20:13.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:20:13.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:20:13.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-13T02:20:13.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:20:13.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:20:13.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:20:13.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:20:13.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:20:13.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:20:13.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:20:13.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:20:13.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:20:13.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:20:13.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:20:13.640 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:20:13.640 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:20:13.640 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:20:13.640 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:20:13.640 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:20:13.640 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:20:13.640 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:20:13.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:20:13.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:20:13.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:20:13.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:20:13.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:20:13.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:20:13.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:20:13.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:20:13.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:20:13.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:20:13.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:20:13.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:20:13.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:20:13.643 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:20:13.644 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a4c7ec00-f0d7-4b7d-816a-72f64986888f 2026-02-13T02:20:13.644 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 a4c7ec00-f0d7-4b7d-816a-72f64986888f' 2026-02-13T02:20:13.644 INFO:tasks.workunit.client.0.trial112.stdout:add osd1 a4c7ec00-f0d7-4b7d-816a-72f64986888f 2026-02-13T02:20:13.645 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:20:13.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBdio5ptr+XJhAAXEi1mLWyde5/7zo83pGTZQ== 2026-02-13T02:20:13.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBdio5ptr+XJhAAXEi1mLWyde5/7zo83pGTZQ=="}' 2026-02-13T02:20:13.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a4c7ec00-f0d7-4b7d-816a-72f64986888f -i td/osd-scrub-repair/1/new.json 2026-02-13T02:20:13.827 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:20:13.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-13T02:20:13.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBdio5ptr+XJhAAXEi1mLWyde5/7zo83pGTZQ== --osd-uuid a4c7ec00-f0d7-4b7d-816a-72f64986888f 2026-02-13T02:20:13.855 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:13.840+0000 7fc22b26aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:13.856 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:13.844+0000 7fc22b26aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:13.857 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:13.844+0000 7fc22b26aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:13.858 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:13.844+0000 7fc22b26aa40 -1 bdev(0x55d98ee3cc00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-13T02:20:13.858 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:13.844+0000 7fc22b26aa40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-13T02:20:14.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-13T02:20:14.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:20:14.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-13T02:20:14.505 INFO:tasks.workunit.client.0.trial112.stdout:adding osd1 key to auth repository 2026-02-13T02:20:14.505 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:20:14.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-13T02:20:14.715 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T02:20:14.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:20:14.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:20:14.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:20:14.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:20:14.739 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:14.724+0000 7f30dd204a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:14.741 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:14.728+0000 7f30dd204a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:14.742 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:14.728+0000 7f30dd204a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:14.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-13T02:20:14.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:20:14.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:20:14.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:20:14.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:20:14.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:20:14.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:20:14.874 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:20:14.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:20:14.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:20:15.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:20:15.142 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:15.128+0000 7f30dd204a40 -1 Falling back to public interface 2026-02-13T02:20:15.686 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:15.672+0000 7f30dd204a40 -1 osd.1 0 log_to_monitors true 2026-02-13T02:20:16.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:20:16.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:20:16.038 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:20:16.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:20:16.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:20:16.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:20:16.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:20:17.206 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:20:17.206 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:20:17.206 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:20:17.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:20:17.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:20:17.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:20:17.371 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3595753845,v1:127.0.0.1:6811/3595753845] [v2:127.0.0.1:6812/3595753845,v1:127.0.0.1:6813/3595753845] exists,up a4c7ec00-f0d7-4b7d-816a-72f64986888f 2026-02-13T02:20:17.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:20:17.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:20:17.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:20:17.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:981: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-13T02:20:17.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:982: list_missing_erasure_coded: run_osd td/osd-scrub-repair 2 2026-02-13T02:20:17.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:20:17.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:20:17.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-13T02:20:17.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:20:17.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-13T02:20:17.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:20:17.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:20:17.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:20:17.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:20:17.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-13T02:20:17.376 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-13T02:20:17.376 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:20:17.376 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:20:17.376 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:20:17.376 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:20:17.376 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:20:17.376 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:20:17.376 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:20:17.376 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:20:17.376 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:20:17.376 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:20:17.376 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:20:17.376 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:20:17.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:20:17.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:20:17.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:20:17.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:20:17.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:20:17.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:20:17.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:20:17.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:20:17.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:20:17.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:20:17.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-13T02:20:17.378 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:20:17.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5b4b015c-7432-45fe-9586-d8118e758cc2 2026-02-13T02:20:17.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 5b4b015c-7432-45fe-9586-d8118e758cc2' 2026-02-13T02:20:17.379 INFO:tasks.workunit.client.0.trial112.stdout:add osd2 5b4b015c-7432-45fe-9586-d8118e758cc2 2026-02-13T02:20:17.379 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:20:17.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBhio5pPuOFFhAAdY/AlFiOeZqClSsGVLPFPQ== 2026-02-13T02:20:17.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBhio5pPuOFFhAAdY/AlFiOeZqClSsGVLPFPQ=="}' 2026-02-13T02:20:17.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5b4b015c-7432-45fe-9586-d8118e758cc2 -i td/osd-scrub-repair/2/new.json 2026-02-13T02:20:17.546 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:20:17.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-13T02:20:17.556 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBhio5pPuOFFhAAdY/AlFiOeZqClSsGVLPFPQ== --osd-uuid 5b4b015c-7432-45fe-9586-d8118e758cc2 2026-02-13T02:20:17.573 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:17.560+0000 7f38578c8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:17.575 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:17.560+0000 7f38578c8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:17.576 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:17.560+0000 7f38578c8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:17.577 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:17.564+0000 7f38578c8a40 -1 bdev(0x560a88e08c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-13T02:20:17.577 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:17.564+0000 7f38578c8a40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-13T02:20:18.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-13T02:20:18.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:20:18.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-13T02:20:18.240 INFO:tasks.workunit.client.0.trial112.stdout:adding osd2 key to auth repository 2026-02-13T02:20:18.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:20:18.442 INFO:tasks.workunit.client.0.trial112.stdout:start osd.2 2026-02-13T02:20:18.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-13T02:20:18.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:20:18.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:20:18.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:20:18.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:20:18.466 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:18.452+0000 7f0f7f66ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:18.468 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:18.452+0000 7f0f7f66ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:18.469 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:18.456+0000 7f0f7f66ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:18.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-13T02:20:18.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:20:18.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T02:20:18.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:20:18.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:20:18.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:20:18.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:20:18.603 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:20:18.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:20:18.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:20:18.764 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:20:18.838 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:18.824+0000 7f0f7f66ca40 -1 Falling back to public interface 2026-02-13T02:20:19.394 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:19.380+0000 7f0f7f66ca40 -1 osd.2 0 log_to_monitors true 2026-02-13T02:20:19.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:20:19.766 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:20:19.766 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:20:19.767 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:20:19.767 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:20:19.767 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:20:19.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:20:20.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:20:20.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:20:20.940 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:20:20.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:20:20.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:20:20.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:20:21.105 INFO:tasks.workunit.client.0.trial112.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4073567973,v1:127.0.0.1:6819/4073567973] [v2:127.0.0.1:6820/4073567973,v1:127.0.0.1:6821/4073567973] exists,up 5b4b015c-7432-45fe-9586-d8118e758cc2 2026-02-13T02:20:21.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:20:21.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:20:21.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:20:21.107 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:984: list_missing_erasure_coded: create_rbd_pool 2026-02-13T02:20:21.107 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-13T02:20:21.261 INFO:tasks.workunit.client.0.trial112.stderr:pool 'rbd' does not exist 2026-02-13T02:20:21.270 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-13T02:20:21.271 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-13T02:20:21.498 INFO:tasks.workunit.client.0.trial112.stderr:pool 'rbd' created 2026-02-13T02:20:21.508 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:20:22.510 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-13T02:20:22.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:985: list_missing_erasure_coded: wait_for_clean 2026-02-13T02:20:22.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:20:22.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:20:22.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:20:22.759 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:20:22.760 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:20:22.760 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:20:22.760 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:20:22.760 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:20:22.760 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:20:22.878 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:20:22.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:20:22.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:20:22.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:20:22.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:20:22.879 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:20:23.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:20:23.040 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:20:23.040 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:20:23.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:20:23.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:20:23.041 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:20:23.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-13T02:20:23.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-13T02:20:23.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-13T02:20:23.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:20:23.115 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:20:23.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705667 2026-02-13T02:20:23.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705667 2026-02-13T02:20:23.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705667' 2026-02-13T02:20:23.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:20:23.189 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:20:23.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542146 2026-02-13T02:20:23.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542146 2026-02-13T02:20:23.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705667 2-60129542146' 2026-02-13T02:20:23.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:20:23.264 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-13T02:20:23.264 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:20:23.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:20:23.266 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-13T02:20:23.266 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:20:23.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-13T02:20:23.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-13T02:20:23.268 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836483 2026-02-13T02:20:23.268 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:20:23.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-13T02:20:23.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:20:24.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:20:24.429 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:20:24.594 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836483 2026-02-13T02:20:24.594 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:20:24.595 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705667 2026-02-13T02:20:24.595 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:20:24.597 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:20:24.598 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705667 2026-02-13T02:20:24.598 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:20:24.600 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705667 2026-02-13T02:20:24.600 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705667 2026-02-13T02:20:24.600 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705667' 2026-02-13T02:20:24.600 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:20:24.760 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705667 -lt 38654705667 2026-02-13T02:20:24.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:20:24.761 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542146 2026-02-13T02:20:24.761 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:20:24.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:20:24.764 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542146 2026-02-13T02:20:24.764 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:20:24.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542146 2026-02-13T02:20:24.766 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542146 2026-02-13T02:20:24.766 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542146' 2026-02-13T02:20:24.766 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:20:24.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542146 -lt 60129542146 2026-02-13T02:20:24.937 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:20:24.938 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:20:24.938 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:20:25.138 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-13T02:20:25.138 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:20:25.139 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:20:25.139 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:20:25.140 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:20:25.140 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:20:25.140 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:20:25.140 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:20:25.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-13T02:20:25.299 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:20:25.300 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:20:25.300 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:20:25.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-13T02:20:25.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:20:25.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:20:25.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:987: list_missing_erasure_coded: create_ec_pool ecpool false k=2 m=1 2026-02-13T02:20:25.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2522: create_ec_pool: local pool_name=ecpool 2026-02-13T02:20:25.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2523: create_ec_pool: shift 2026-02-13T02:20:25.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2524: create_ec_pool: local allow_overwrites=false 2026-02-13T02:20:25.494 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2525: create_ec_pool: shift 2026-02-13T02:20:25.494 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2527: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2026-02-13T02:20:25.698 INFO:tasks.workunit.client.0.trial112.stderr:load: isa 2026-02-13T02:20:25.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2529: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2026-02-13T02:20:25.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2026-02-13T02:20:26.004 INFO:tasks.workunit.client.0.trial112.stderr:pool 'ecpool' created 2026-02-13T02:20:26.017 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:20:27.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2531: create_ec_pool: '[' false = true ']' 2026-02-13T02:20:27.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2535: create_ec_pool: wait_for_clean 2026-02-13T02:20:27.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:20:27.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:20:27.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:20:27.019 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:20:27.020 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:20:27.020 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:20:27.021 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:20:27.021 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:20:27.021 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:20:27.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:20:27.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:20:27.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:20:27.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:20:27.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:20:27.143 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:20:27.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:20:27.301 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:20:27.301 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:20:27.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:20:27.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:20:27.301 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:20:27.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836485 2026-02-13T02:20:27.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836485 2026-02-13T02:20:27.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485' 2026-02-13T02:20:27.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:20:27.375 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:20:27.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705669 2026-02-13T02:20:27.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705669 2026-02-13T02:20:27.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-38654705669' 2026-02-13T02:20:27.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:20:27.449 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:20:27.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542148 2026-02-13T02:20:27.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542148 2026-02-13T02:20:27.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-38654705669 2-60129542148' 2026-02-13T02:20:27.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:20:27.521 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836485 2026-02-13T02:20:27.521 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:20:27.523 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:20:27.524 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836485 2026-02-13T02:20:27.524 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:20:27.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836485 2026-02-13T02:20:27.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-13T02:20:27.526 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836485 2026-02-13T02:20:27.526 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:20:27.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836485 2026-02-13T02:20:27.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:20:28.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:20:28.687 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:20:28.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 21474836485 2026-02-13T02:20:28.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:20:28.852 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705669 2026-02-13T02:20:28.852 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:20:28.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:20:28.854 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705669 2026-02-13T02:20:28.855 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:20:28.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705669 2026-02-13T02:20:28.856 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705669 2026-02-13T02:20:28.857 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705669' 2026-02-13T02:20:28.857 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:20:29.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705669 -lt 38654705669 2026-02-13T02:20:29.023 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:20:29.023 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542148 2026-02-13T02:20:29.024 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:20:29.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:20:29.026 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542148 2026-02-13T02:20:29.026 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:20:29.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542148 2026-02-13T02:20:29.028 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542148 2026-02-13T02:20:29.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542148' 2026-02-13T02:20:29.028 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:20:29.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542148 -lt 60129542148 2026-02-13T02:20:29.197 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:20:29.198 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:20:29.198 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:20:29.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:20:29.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:20:29.392 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:20:29.392 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:20:29.392 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:20:29.392 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:20:29.393 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:20:29.393 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:20:29.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:20:29.549 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:20:29.550 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:20:29.550 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:20:29.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:20:29.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:20:29.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:20:29.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2536: create_ec_pool: return 0 2026-02-13T02:20:29.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:990: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ0 2026-02-13T02:20:29.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:20:29.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-13T02:20:29.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ0 2026-02-13T02:20:29.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:20:29.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:20:29.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:20:29.945 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:20:29.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:20:30.150 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:20:30.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:20:30.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:20:30.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ0 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:20:30.185 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:991: list_missing_erasure_coded: get_osds ecpool MOBJ0 2026-02-13T02:20:30.186 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2026-02-13T02:20:30.186 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=MOBJ0 2026-02-13T02:20:30.186 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool MOBJ0 2026-02-13T02:20:30.186 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-13T02:20:30.352 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=2 2026-02-13T02:20:30.352 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:20:30.353 INFO:tasks.workunit.client.0.trial112.stderr:0' 2026-02-13T02:20:30.353 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 2 1 0 2026-02-13T02:20:30.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:991: list_missing_erasure_coded: osds0=('2' '1' '0') 2026-02-13T02:20:30.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:991: list_missing_erasure_coded: local -a osds0 2026-02-13T02:20:30.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:994: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ1 2026-02-13T02:20:30.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:20:30.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-13T02:20:30.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ1 2026-02-13T02:20:30.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:20:30.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:20:30.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:20:30.555 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:20:30.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:20:30.758 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:20:30.770 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:20:30.770 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:20:30.770 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ1 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:20:30.791 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:995: list_missing_erasure_coded: get_osds ecpool MOBJ1 2026-02-13T02:20:30.791 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2026-02-13T02:20:30.791 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=MOBJ1 2026-02-13T02:20:30.792 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool MOBJ1 2026-02-13T02:20:30.792 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-13T02:20:30.949 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=2 2026-02-13T02:20:30.949 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:20:30.949 INFO:tasks.workunit.client.0.trial112.stderr:0' 2026-02-13T02:20:30.949 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 2 1 0 2026-02-13T02:20:30.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:995: list_missing_erasure_coded: osds1=('2' '1' '0') 2026-02-13T02:20:30.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:995: list_missing_erasure_coded: local -a osds1 2026-02-13T02:20:30.949 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:998: list_missing_erasure_coded: seq 0 2 2026-02-13T02:20:30.950 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:998: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-13T02:20:30.950 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:999: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T02:20:30.951 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:20:30.951 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:20:30.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:20:30.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:20:30.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:20:31.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:20:31.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:998: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-13T02:20:31.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:999: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:20:31.263 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:20:31.263 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:20:31.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:20:31.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:20:31.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:20:31.574 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:20:31.574 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:998: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-13T02:20:31.574 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:999: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.2 2026-02-13T02:20:31.575 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:20:31.576 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:20:31.576 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:20:31.576 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:20:31.576 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:20:31.886 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:20:31.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1002: list_missing_erasure_coded: id=2 2026-02-13T02:20:31.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1003: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 MOBJ0 remove 2026-02-13T02:20:32.081 INFO:tasks.workunit.client.0.trial112.stdout:remove 0#2:a9030505:::MOBJ0:head# 2026-02-13T02:20:32.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1005: list_missing_erasure_coded: id=1 2026-02-13T02:20:32.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ0 remove 2026-02-13T02:20:32.361 INFO:tasks.workunit.client.0.trial112.stdout:remove 1#2:a9030505:::MOBJ0:head# 2026-02-13T02:20:32.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1009: list_missing_erasure_coded: id=1 2026-02-13T02:20:32.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ1 remove 2026-02-13T02:20:32.620 INFO:tasks.workunit.client.0.trial112.stdout:remove 1#2:8dd16d55:::MOBJ1:head# 2026-02-13T02:20:32.704 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1012: list_missing_erasure_coded: id=0 2026-02-13T02:20:32.705 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1013: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 MOBJ1 remove 2026-02-13T02:20:32.901 INFO:tasks.workunit.client.0.trial112.stdout:remove 2#2:8dd16d55:::MOBJ1:head# 2026-02-13T02:20:32.992 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: list_missing_erasure_coded: seq 0 2 2026-02-13T02:20:32.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-13T02:20:32.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1017: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 0 2026-02-13T02:20:32.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:20:32.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:20:32.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:20:32.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:20:32.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:20:32.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:20:32.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:20:32.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:20:32.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:20:32.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:20:32.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:20:32.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:20:32.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:20:32.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:20:32.996 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:20:32.996 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:20:32.996 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:20:32.996 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:20:32.997 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:20:32.997 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:20:32.998 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:20:32.998 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:20:32.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:20:32.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:20:32.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:20:32.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:20:32.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:20:32.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:20:32.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:20:32.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:20:32.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:20:32.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:20:32.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:20:33.000 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T02:20:33.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:20:33.000 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T02:20:33.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:20:33.001 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:20:33.001 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:20:33.001 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:20:33.017 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:33.003+0000 7fb6826e5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:33.019 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:33.003+0000 7fb6826e5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:33.020 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:33.003+0000 7fb6826e5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:33.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:20:33.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:20:33.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:20:33.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:20:33.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:20:33.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:20:33.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:20:33.157 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:20:33.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:20:33.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:20:33.319 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:20:33.394 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:33.379+0000 7fb6826e5a40 -1 Falling back to public interface 2026-02-13T02:20:33.939 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:33.923+0000 7fb6826e5a40 -1 osd.0 31 log_to_monitors true 2026-02-13T02:20:34.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:20:34.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:20:34.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:20:34.321 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:20:34.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:20:34.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:20:34.487 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 38 up_thru 38 down_at 32 last_clean_interval [5,31) [v2:127.0.0.1:6800/2804387129,v1:127.0.0.1:6801/2804387129] [v2:127.0.0.1:6802/2804387129,v1:127.0.0.1:6803/2804387129] exists,up 25c08f3b-b1f6-4e4d-b941-e57ea07c391e 2026-02-13T02:20:34.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:20:34.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:20:34.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:20:34.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-13T02:20:34.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1017: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 1 2026-02-13T02:20:34.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:20:34.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:20:34.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:20:34.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:20:34.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:20:34.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:20:34.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:20:34.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:20:34.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:20:34.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:20:34.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:20:34.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:20:34.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:20:34.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:20:34.489 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:20:34.489 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:20:34.489 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:20:34.490 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:20:34.490 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:20:34.490 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:20:34.490 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:20:34.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:20:34.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:20:34.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:20:34.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:20:34.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:20:34.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:20:34.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:20:34.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:20:34.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:20:34.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:20:34.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:20:34.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:20:34.491 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:20:34.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:20:34.491 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:20:34.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:20:34.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:20:34.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:20:34.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:20:34.511 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:34.495+0000 7f577e717a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:34.513 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:34.499+0000 7f577e717a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:34.514 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:34.499+0000 7f577e717a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:34.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:20:34.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:20:34.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:20:34.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:20:34.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:20:34.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:20:34.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:20:34.652 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:20:34.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:20:34.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:20:34.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:20:34.942 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:34.927+0000 7f577e717a40 -1 Falling back to public interface 2026-02-13T02:20:35.587 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:35.571+0000 7f577e717a40 -1 osd.1 33 log_to_monitors true 2026-02-13T02:20:35.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:20:35.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:20:35.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:20:35.819 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:20:35.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:20:35.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:20:35.986 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 42 up_thru 42 down_at 34 last_clean_interval [9,33) [v2:127.0.0.1:6810/858908428,v1:127.0.0.1:6811/858908428] [v2:127.0.0.1:6812/858908428,v1:127.0.0.1:6813/858908428] exists,up a4c7ec00-f0d7-4b7d-816a-72f64986888f 2026-02-13T02:20:35.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:20:35.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:20:35.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:20:35.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-13T02:20:35.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1017: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 2 2026-02-13T02:20:35.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:20:35.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:20:35.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-13T02:20:35.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:20:35.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2026-02-13T02:20:35.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:20:35.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:20:35.988 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:20:35.988 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:20:35.988 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-13T02:20:35.988 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-13T02:20:35.988 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:20:35.988 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:20:35.988 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:20:35.988 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:20:35.988 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:20:35.988 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:20:35.989 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:20:35.989 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:20:35.989 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:20:35.989 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:20:35.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:20:35.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:20:35.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:20:35.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:20:35.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:20:35.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:20:35.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:20:35.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:20:35.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:20:35.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:20:35.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2026-02-13T02:20:35.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-13T02:20:35.990 INFO:tasks.workunit.client.0.trial112.stderr:start osd.2 2026-02-13T02:20:35.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:20:35.991 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2026-02-13T02:20:35.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-13T02:20:35.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:20:35.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:20:35.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:20:36.003 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:35.987+0000 7f701ce31a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:36.004 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:35.987+0000 7f701ce31a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:36.005 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:35.991+0000 7f701ce31a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:36.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-13T02:20:36.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:20:36.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T02:20:36.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:20:36.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:20:36.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:20:36.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:20:36.148 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:20:36.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:20:36.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:20:36.315 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:20:36.450 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:36.435+0000 7f701ce31a40 -1 Falling back to public interface 2026-02-13T02:20:37.110 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:37.095+0000 7f701ce31a40 -1 osd.2 35 log_to_monitors true 2026-02-13T02:20:37.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:20:37.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:20:37.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:20:37.318 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:20:37.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:20:37.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:20:37.479 INFO:tasks.workunit.client.0.trial112.stderr:osd.2 up in weight 1 up_from 46 up_thru 46 down_at 36 last_clean_interval [14,35) [v2:127.0.0.1:6818/1662767119,v1:127.0.0.1:6819/1662767119] [v2:127.0.0.1:6820/1662767119,v1:127.0.0.1:6821/1662767119] exists,up 5b4b015c-7432-45fe-9586-d8118e758cc2 2026-02-13T02:20:37.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:20:37.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:20:37.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:20:37.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: list_missing_erasure_coded: create_rbd_pool 2026-02-13T02:20:37.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-13T02:20:37.683 INFO:tasks.workunit.client.0.trial112.stderr:pool 'rbd' removed 2026-02-13T02:20:37.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-13T02:20:37.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-13T02:20:37.887 INFO:tasks.workunit.client.0.trial112.stderr:pool 'rbd' created 2026-02-13T02:20:37.898 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:20:38.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-13T02:20:39.183 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: list_missing_erasure_coded: wait_for_clean 2026-02-13T02:20:39.183 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:20:39.183 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:20:39.183 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:20:39.184 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:20:39.184 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:20:39.184 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:20:39.185 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:20:39.185 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:20:39.185 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:20:39.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:20:39.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:20:39.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:20:39.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:20:39.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:20:39.303 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:20:39.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:20:39.466 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:20:39.467 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:20:39.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:20:39.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:20:39.467 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:20:39.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=163208757251 2026-02-13T02:20:39.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 163208757251 2026-02-13T02:20:39.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-163208757251' 2026-02-13T02:20:39.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:20:39.544 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:20:39.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626434 2026-02-13T02:20:39.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626434 2026-02-13T02:20:39.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-163208757251 1-180388626434' 2026-02-13T02:20:39.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:20:39.617 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:20:39.692 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=197568495618 2026-02-13T02:20:39.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 197568495618 2026-02-13T02:20:39.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-163208757251 1-180388626434 2-197568495618' 2026-02-13T02:20:39.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:20:39.693 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-163208757251 2026-02-13T02:20:39.694 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:20:39.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:20:39.696 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-163208757251 2026-02-13T02:20:39.696 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:20:39.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=163208757251 2026-02-13T02:20:39.698 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 163208757251 2026-02-13T02:20:39.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 163208757251' 2026-02-13T02:20:39.698 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:20:39.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 163208757251 2026-02-13T02:20:39.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:20:40.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:20:40.862 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:20:41.024 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 163208757251 -lt 163208757251 2026-02-13T02:20:41.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:20:41.025 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-180388626434 2026-02-13T02:20:41.025 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:20:41.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:20:41.028 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-180388626434 2026-02-13T02:20:41.028 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:20:41.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626434 2026-02-13T02:20:41.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 180388626434' 2026-02-13T02:20:41.030 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 180388626434 2026-02-13T02:20:41.030 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:20:41.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626434 -lt 180388626434 2026-02-13T02:20:41.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:20:41.193 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-197568495618 2026-02-13T02:20:41.193 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:20:41.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:20:41.195 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-197568495618 2026-02-13T02:20:41.196 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:20:41.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=197568495618 2026-02-13T02:20:41.197 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 197568495618 2026-02-13T02:20:41.198 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 197568495618' 2026-02-13T02:20:41.198 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:20:41.372 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 197568495618 -lt 197568495618 2026-02-13T02:20:41.373 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:20:41.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:20:41.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:20:41.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:20:41.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:20:41.572 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:20:41.572 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:20:41.572 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:20:41.572 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:20:41.572 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:20:41.572 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:20:41.729 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:20:41.730 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:20:41.730 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:20:41.730 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:20:41.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:20:41.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:20:41.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:20:41.924 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: list_missing_erasure_coded: get_pg ecpool MOBJ0 2026-02-13T02:20:41.924 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-13T02:20:41.924 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=MOBJ0 2026-02-13T02:20:41.924 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool MOBJ0 2026-02-13T02:20:41.924 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T02:20:42.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: list_missing_erasure_coded: local pg=2.0 2026-02-13T02:20:42.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: list_missing_erasure_coded: repair 2.0 2026-02-13T02:20:42.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-13T02:20:42.084 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-13T02:20:42.084 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:20:42.084 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:20:42.085 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:20:42.085 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:20:42.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-13T02:20:25.993425+0000 2026-02-13T02:20:42.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-13T02:20:42.398 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 2.0s0 on osd.2 to repair 2026-02-13T02:20:42.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-13T02:20:25.993425+0000 2026-02-13T02:20:42.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-13T02:20:42.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:20:25.993425+0000 2026-02-13T02:20:42.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:20:42.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:20:42.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:20:42.409 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:20:42.409 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:20:42.409 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:20:42.409 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:20:42.410 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:20:42.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:20:25.993425+0000 '>' 2026-02-13T02:20:25.993425+0000 2026-02-13T02:20:42.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:20:43.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:20:43.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:20:43.572 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:20:43.572 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:20:43.572 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:20:43.572 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:20:43.572 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:20:43.729 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:20:25.993425+0000 '>' 2026-02-13T02:20:25.993425+0000 2026-02-13T02:20:43.729 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:20:44.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:20:44.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:20:44.731 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:20:44.731 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:20:44.731 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:20:44.732 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:20:44.732 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:20:44.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:20:25.993425+0000 '>' 2026-02-13T02:20:25.993425+0000 2026-02-13T02:20:44.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:20:45.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:20:45.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:20:45.891 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:20:45.891 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:20:45.891 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:20:45.892 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:20:45.892 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:20:46.050 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:20:25.993425+0000 '>' 2026-02-13T02:20:25.993425+0000 2026-02-13T02:20:46.051 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:20:47.052 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:20:47.052 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:20:47.053 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:20:47.053 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:20:47.053 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:20:47.053 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:20:47.054 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:20:47.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:20:25.993425+0000 '>' 2026-02-13T02:20:25.993425+0000 2026-02-13T02:20:47.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:20:48.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:20:48.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:20:48.216 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:20:48.216 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:20:48.216 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:20:48.216 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:20:48.216 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:20:48.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:20:25.993425+0000 '>' 2026-02-13T02:20:25.993425+0000 2026-02-13T02:20:48.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:20:49.372 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:20:49.372 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:20:49.373 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:20:49.373 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:20:49.373 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:20:49.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:20:49.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:20:49.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:20:43.097297+0000 '>' 2026-02-13T02:20:25.993425+0000 2026-02-13T02:20:49.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:20:49.534 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: list_missing_erasure_coded: seq 0 120 2026-02-13T02:20:49.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: list_missing_erasure_coded: for i in $(seq 0 120) 2026-02-13T02:20:49.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: list_missing_erasure_coded: '[' 0 -lt 60 ']' 2026-02-13T02:20:49.535 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1031: list_missing_erasure_coded: ceph pg 2.0 list_unfound 2026-02-13T02:20:49.536 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1031: list_missing_erasure_coded: egrep 'MOBJ0|MOBJ1' 2026-02-13T02:20:49.536 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1031: list_missing_erasure_coded: wc -l 2026-02-13T02:20:49.611 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1031: list_missing_erasure_coded: matches=2 2026-02-13T02:20:49.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1032: list_missing_erasure_coded: '[' 2 -eq 2 ']' 2026-02-13T02:20:49.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1032: list_missing_erasure_coded: break 2026-02-13T02:20:49.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-13T02:20:49.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:20:49.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:20:49.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:20:49.612 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:20:49.612 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:20:49.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:20:49.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:20:49.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:20:49.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:20:49.924 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:20:49.925 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:20:49.926 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:20:49.928 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:20:49.928 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:20:49.929 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:20:49.929 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:20:49.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:20:49.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:20:49.930 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:20:49.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:20:49.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:20:49.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:20:49.948 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:20:49.948 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:20:49.948 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:20:49.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:20:49.950 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:20:49.950 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:20:49.950 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-13T02:20:49.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-13T02:20:49.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-13T02:20:49.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-13T02:20:49.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:20:49.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:20:49.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:20:49.952 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:20:49.952 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:20:49.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:20:49.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:20:49.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:20:49.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:20:49.955 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:20:49.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:20:49.957 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:20:49.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:20:49.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:20:49.958 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:20:49.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:20:49.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:20:49.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:20:49.961 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:20:49.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:20:49.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:20:49.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:20:49.964 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:20:49.964 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:20:49.964 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:20:49.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:20:49.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:20:49.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:20:49.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-13T02:20:49.967 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T02:20:49.967 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:20:49.968 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:20:49.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.35207 2026-02-13T02:20:49.969 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T02:20:49.971 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-13T02:20:49.971 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T02:20:49.971 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-13T02:20:49.971 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_list_missing_erasure_coded_overwrites td/osd-scrub-repair 2026-02-13T02:20:49.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1041: TEST_list_missing_erasure_coded_overwrites: '[' true = true ']' 2026-02-13T02:20:49.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1042: TEST_list_missing_erasure_coded_overwrites: list_missing_erasure_coded td/osd-scrub-repair true 2026-02-13T02:20:49.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:975: list_missing_erasure_coded: local dir=td/osd-scrub-repair 2026-02-13T02:20:49.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:976: list_missing_erasure_coded: local allow_overwrites=true 2026-02-13T02:20:49.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:977: list_missing_erasure_coded: local poolname=ecpool 2026-02-13T02:20:49.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:979: list_missing_erasure_coded: run_mon td/osd-scrub-repair a 2026-02-13T02:20:49.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-13T02:20:49.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-13T02:20:49.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-13T02:20:49.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-13T02:20:49.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-13T02:20:49.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-13T02:20:49.992 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-13T02:20:49.992 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:20:49.993 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:20:49.994 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:20:49.994 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:20:49.994 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:20:49.994 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:20:49.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-13T02:20:50.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-13T02:20:50.021 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-13T02:20:50.022 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:20:50.022 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:20:50.022 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-13T02:20:50.023 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-13T02:20:50.023 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:20:50.023 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:20:50.023 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:20:50.023 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:20:50.023 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:20:50.023 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:20:50.024 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:20:50.024 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:20:50.024 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get fsid 2026-02-13T02:20:50.079 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-13T02:20:50.079 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:20:50.079 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:20:50.079 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-13T02:20:50.079 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-13T02:20:50.081 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:20:50.081 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:20:50.081 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:20:50.081 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:20:50.081 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:20:50.081 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:20:50.081 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:20:50.082 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:20:50.082 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get mon_host 2026-02-13T02:20:50.136 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:980: list_missing_erasure_coded: run_mgr td/osd-scrub-repair x 2026-02-13T02:20:50.136 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-13T02:20:50.136 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-13T02:20:50.136 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-13T02:20:50.136 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-13T02:20:50.136 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-13T02:20:50.136 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-13T02:20:50.230 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-13T02:20:50.230 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:20:50.230 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:20:50.230 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:20:50.230 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:20:50.230 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:20:50.231 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:20:50.231 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:20:50.232 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:20:50.253 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:981: list_missing_erasure_coded: seq 0 2 2026-02-13T02:20:50.255 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:981: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-13T02:20:50.255 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:982: list_missing_erasure_coded: run_osd td/osd-scrub-repair 0 2026-02-13T02:20:50.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:20:50.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:20:50.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-13T02:20:50.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:20:50.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:20:50.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:20:50.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:20:50.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:20:50.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:20:50.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:20:50.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:20:50.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:20:50.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:20:50.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:20:50.257 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:20:50.257 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:20:50.257 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:20:50.257 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:20:50.257 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:20:50.257 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:20:50.259 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:20:50.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:20:50.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:20:50.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:20:50.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:20:50.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:20:50.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:20:50.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:20:50.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:20:50.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:20:50.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:20:50.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:20:50.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:20:50.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:20:50.260 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:20:50.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=bed6d058-d5fb-4a47-861a-bc1e403ce73b 2026-02-13T02:20:50.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 bed6d058-d5fb-4a47-861a-bc1e403ce73b' 2026-02-13T02:20:50.261 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 bed6d058-d5fb-4a47-861a-bc1e403ce73b 2026-02-13T02:20:50.261 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:20:50.269 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCCio5pvv5xDxAAGmBJcXlljROLJgtCsVH7aQ== 2026-02-13T02:20:50.270 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCCio5pvv5xDxAAGmBJcXlljROLJgtCsVH7aQ=="}' 2026-02-13T02:20:50.270 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new bed6d058-d5fb-4a47-861a-bc1e403ce73b -i td/osd-scrub-repair/0/new.json 2026-02-13T02:20:50.363 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:20:50.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-13T02:20:50.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCCio5pvv5xDxAAGmBJcXlljROLJgtCsVH7aQ== --osd-uuid bed6d058-d5fb-4a47-861a-bc1e403ce73b 2026-02-13T02:20:50.389 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:50.375+0000 7ff4f9253a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:50.391 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:50.375+0000 7ff4f9253a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:50.392 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:50.375+0000 7ff4f9253a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:50.392 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:50.375+0000 7ff4f9253a40 -1 bdev(0x557e73502c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-13T02:20:50.393 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:50.379+0000 7ff4f9253a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-13T02:20:51.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-13T02:20:51.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:20:51.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-13T02:20:51.062 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T02:20:51.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:20:51.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-13T02:20:51.165 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T02:20:51.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:20:51.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:20:51.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:20:51.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:20:51.188 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:51.171+0000 7f1239784a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:51.190 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:51.175+0000 7f1239784a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:51.192 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:51.175+0000 7f1239784a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:51.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-13T02:20:51.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:20:51.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:20:51.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:20:51.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:20:51.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:20:51.260 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:20:51.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:20:51.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:20:51.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:20:51.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:20:51.686 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:51.671+0000 7f1239784a40 -1 Falling back to public interface 2026-02-13T02:20:52.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:20:52.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:20:52.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:20:52.351 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:20:52.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:20:52.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:20:52.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:20:52.626 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:52.611+0000 7f1239784a40 -1 osd.0 0 log_to_monitors true 2026-02-13T02:20:53.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:20:53.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:20:53.472 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:20:53.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:20:53.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:20:53.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:20:53.630 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3041261902,v1:127.0.0.1:6801/3041261902] [v2:127.0.0.1:6802/3041261902,v1:127.0.0.1:6803/3041261902] exists,up bed6d058-d5fb-4a47-861a-bc1e403ce73b 2026-02-13T02:20:53.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:20:53.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:20:53.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:20:53.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:981: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-13T02:20:53.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:982: list_missing_erasure_coded: run_osd td/osd-scrub-repair 1 2026-02-13T02:20:53.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:20:53.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:20:53.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-13T02:20:53.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:20:53.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:20:53.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:20:53.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:20:53.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:20:53.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:20:53.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:20:53.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:20:53.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:20:53.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:20:53.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:20:53.635 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:20:53.635 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:20:53.635 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:20:53.635 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:20:53.636 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:20:53.636 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:20:53.636 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:20:53.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:20:53.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:20:53.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:20:53.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:20:53.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:20:53.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:20:53.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:20:53.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:20:53.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:20:53.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:20:53.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:20:53.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:20:53.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:20:53.637 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:20:53.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8984e199-f50d-43c5-98e5-cae80770f4fb 2026-02-13T02:20:53.638 INFO:tasks.workunit.client.0.trial112.stdout:add osd1 8984e199-f50d-43c5-98e5-cae80770f4fb 2026-02-13T02:20:53.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 8984e199-f50d-43c5-98e5-cae80770f4fb' 2026-02-13T02:20:53.638 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:20:53.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCFio5p7NoFJhAAqHWLkfDXOnNLhMEm1v0i6Q== 2026-02-13T02:20:53.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCFio5p7NoFJhAAqHWLkfDXOnNLhMEm1v0i6Q=="}' 2026-02-13T02:20:53.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8984e199-f50d-43c5-98e5-cae80770f4fb -i td/osd-scrub-repair/1/new.json 2026-02-13T02:20:53.804 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:20:53.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-13T02:20:53.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCFio5p7NoFJhAAqHWLkfDXOnNLhMEm1v0i6Q== --osd-uuid 8984e199-f50d-43c5-98e5-cae80770f4fb 2026-02-13T02:20:53.831 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:53.815+0000 7f36519d0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:53.832 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:53.819+0000 7f36519d0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:53.834 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:53.819+0000 7f36519d0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:53.834 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:53.819+0000 7f36519d0a40 -1 bdev(0x55c38c2dec00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-13T02:20:53.834 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:53.819+0000 7f36519d0a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-13T02:20:54.455 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-13T02:20:54.455 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:20:54.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-13T02:20:54.457 INFO:tasks.workunit.client.0.trial112.stdout:adding osd1 key to auth repository 2026-02-13T02:20:54.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:20:54.652 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T02:20:54.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-13T02:20:54.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:20:54.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:20:54.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:20:54.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:20:54.676 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:54.659+0000 7f8ed3d6fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:54.678 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:54.663+0000 7f8ed3d6fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:54.679 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:54.663+0000 7f8ed3d6fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:54.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-13T02:20:54.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:20:54.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:20:54.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:20:54.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:20:54.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:20:54.814 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:20:54.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:20:54.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:20:54.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:20:54.971 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:20:55.090 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:55.075+0000 7f8ed3d6fa40 -1 Falling back to public interface 2026-02-13T02:20:55.686 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:55.671+0000 7f8ed3d6fa40 -1 osd.1 0 log_to_monitors true 2026-02-13T02:20:55.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:20:55.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:20:55.974 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:20:55.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:20:55.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:20:55.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:20:56.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:20:57.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:20:57.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:20:57.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:20:57.136 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:20:57.136 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:20:57.136 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:20:57.292 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2604023917,v1:127.0.0.1:6811/2604023917] [v2:127.0.0.1:6812/2604023917,v1:127.0.0.1:6813/2604023917] exists,up 8984e199-f50d-43c5-98e5-cae80770f4fb 2026-02-13T02:20:57.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:20:57.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:20:57.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:20:57.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:981: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-13T02:20:57.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:982: list_missing_erasure_coded: run_osd td/osd-scrub-repair 2 2026-02-13T02:20:57.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:20:57.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:20:57.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-13T02:20:57.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:20:57.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-13T02:20:57.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:20:57.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:20:57.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:20:57.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:20:57.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-13T02:20:57.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-13T02:20:57.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:20:57.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:20:57.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:20:57.297 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:20:57.298 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:20:57.298 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:20:57.298 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:20:57.298 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:20:57.298 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:20:57.298 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:20:57.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:20:57.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:20:57.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:20:57.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:20:57.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:20:57.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:20:57.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:20:57.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:20:57.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:20:57.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:20:57.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:20:57.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:20:57.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-13T02:20:57.300 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:20:57.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=872361e8-6578-41a3-a84e-79a9afcabd6a 2026-02-13T02:20:57.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 872361e8-6578-41a3-a84e-79a9afcabd6a' 2026-02-13T02:20:57.301 INFO:tasks.workunit.client.0.trial112.stdout:add osd2 872361e8-6578-41a3-a84e-79a9afcabd6a 2026-02-13T02:20:57.301 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:20:57.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCJio5pZQ3mERAAfuwAE6nMRw67yjlbfyrfCw== 2026-02-13T02:20:57.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCJio5pZQ3mERAAfuwAE6nMRw67yjlbfyrfCw=="}' 2026-02-13T02:20:57.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 872361e8-6578-41a3-a84e-79a9afcabd6a -i td/osd-scrub-repair/2/new.json 2026-02-13T02:20:57.473 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:20:57.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-13T02:20:57.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCJio5pZQ3mERAAfuwAE6nMRw67yjlbfyrfCw== --osd-uuid 872361e8-6578-41a3-a84e-79a9afcabd6a 2026-02-13T02:20:57.499 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:57.483+0000 7f421b394a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:57.500 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:57.483+0000 7f421b394a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:57.501 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:57.487+0000 7f421b394a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:57.502 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:57.487+0000 7f421b394a40 -1 bdev(0x556263334c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-13T02:20:57.502 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:57.487+0000 7f421b394a40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-13T02:20:58.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-13T02:20:58.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:20:58.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-13T02:20:58.142 INFO:tasks.workunit.client.0.trial112.stdout:adding osd2 key to auth repository 2026-02-13T02:20:58.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:20:58.347 INFO:tasks.workunit.client.0.trial112.stdout:start osd.2 2026-02-13T02:20:58.348 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-13T02:20:58.348 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:20:58.348 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:20:58.348 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:20:58.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:20:58.372 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:58.355+0000 7ff4e338fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:58.374 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:58.359+0000 7ff4e338fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:58.375 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:58.359+0000 7ff4e338fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:20:58.510 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-13T02:20:58.510 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:20:58.510 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T02:20:58.510 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:20:58.510 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:20:58.510 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:20:58.511 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:20:58.511 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:20:58.512 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:20:58.512 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:20:58.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:20:58.762 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:58.747+0000 7ff4e338fa40 -1 Falling back to public interface 2026-02-13T02:20:59.318 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:20:59.303+0000 7ff4e338fa40 -1 osd.2 0 log_to_monitors true 2026-02-13T02:20:59.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:20:59.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:20:59.669 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:20:59.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:20:59.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:20:59.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:20:59.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:21:00.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:21:00.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:21:00.832 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:21:00.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:21:00.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:21:00.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:21:00.996 INFO:tasks.workunit.client.0.trial112.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/383806916,v1:127.0.0.1:6819/383806916] [v2:127.0.0.1:6820/383806916,v1:127.0.0.1:6821/383806916] exists,up 872361e8-6578-41a3-a84e-79a9afcabd6a 2026-02-13T02:21:00.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:21:00.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:21:00.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:21:00.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:984: list_missing_erasure_coded: create_rbd_pool 2026-02-13T02:21:00.998 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-13T02:21:01.151 INFO:tasks.workunit.client.0.trial112.stderr:pool 'rbd' does not exist 2026-02-13T02:21:01.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-13T02:21:01.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-13T02:21:01.405 INFO:tasks.workunit.client.0.trial112.stderr:pool 'rbd' created 2026-02-13T02:21:01.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:21:02.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-13T02:21:02.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:985: list_missing_erasure_coded: wait_for_clean 2026-02-13T02:21:02.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:21:02.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:21:02.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:21:02.684 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:21:02.685 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:21:02.685 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:21:02.685 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:21:02.685 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:21:02.685 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:21:02.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:21:02.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:21:02.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:21:02.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:21:02.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:21:02.801 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:21:02.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:21:02.962 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:21:02.962 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:21:02.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:21:02.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:21:02.963 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:21:03.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-13T02:21:03.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-13T02:21:03.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-13T02:21:03.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:21:03.036 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:21:03.107 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705668 2026-02-13T02:21:03.108 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705668 2026-02-13T02:21:03.108 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705668' 2026-02-13T02:21:03.108 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:21:03.108 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:21:03.181 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574850 2026-02-13T02:21:03.181 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574850 2026-02-13T02:21:03.181 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705668 2-55834574850' 2026-02-13T02:21:03.182 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:21:03.182 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-13T02:21:03.182 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:21:03.184 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:21:03.185 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-13T02:21:03.185 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:21:03.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-13T02:21:03.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-13T02:21:03.187 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836483 2026-02-13T02:21:03.187 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:21:03.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-13T02:21:03.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:21:04.357 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:21:04.358 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:21:04.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-13T02:21:04.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:21:05.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:21:05.522 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:21:05.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836483 2026-02-13T02:21:05.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:21:05.685 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705668 2026-02-13T02:21:05.686 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:21:05.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:21:05.689 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705668 2026-02-13T02:21:05.689 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:21:05.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705668 2026-02-13T02:21:05.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705668' 2026-02-13T02:21:05.690 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705668 2026-02-13T02:21:05.691 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:21:05.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705668 -lt 38654705668 2026-02-13T02:21:05.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:21:05.863 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574850 2026-02-13T02:21:05.864 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:21:05.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:21:05.866 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574850 2026-02-13T02:21:05.866 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:21:05.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574850 2026-02-13T02:21:05.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574850' 2026-02-13T02:21:05.868 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574850 2026-02-13T02:21:05.868 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:21:06.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574850 -lt 55834574850 2026-02-13T02:21:06.039 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:21:06.040 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:21:06.040 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:21:06.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-13T02:21:06.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:21:06.239 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:21:06.239 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:21:06.240 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:21:06.240 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:21:06.240 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:21:06.240 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:21:06.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-13T02:21:06.401 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:21:06.401 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:21:06.401 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:21:06.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-13T02:21:06.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:21:06.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:21:06.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:987: list_missing_erasure_coded: create_ec_pool ecpool true k=2 m=1 2026-02-13T02:21:06.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2522: create_ec_pool: local pool_name=ecpool 2026-02-13T02:21:06.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2523: create_ec_pool: shift 2026-02-13T02:21:06.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2524: create_ec_pool: local allow_overwrites=true 2026-02-13T02:21:06.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2525: create_ec_pool: shift 2026-02-13T02:21:06.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2527: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2026-02-13T02:21:06.806 INFO:tasks.workunit.client.0.trial112.stderr:load: isa 2026-02-13T02:21:06.816 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2529: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2026-02-13T02:21:06.816 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2026-02-13T02:21:07.111 INFO:tasks.workunit.client.0.trial112.stderr:pool 'ecpool' created 2026-02-13T02:21:07.125 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:21:08.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2531: create_ec_pool: '[' true = true ']' 2026-02-13T02:21:08.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2532: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2026-02-13T02:21:08.331 INFO:tasks.workunit.client.0.trial112.stderr:set pool 2 allow_ec_overwrites to true 2026-02-13T02:21:08.343 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2535: create_ec_pool: wait_for_clean 2026-02-13T02:21:08.343 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:21:08.343 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:21:08.343 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:21:08.344 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:21:08.344 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:21:08.344 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:21:08.345 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:21:08.345 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:21:08.345 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:21:08.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:21:08.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:21:08.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:21:08.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:21:08.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:21:08.473 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:21:08.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:21:08.639 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:21:08.639 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:21:08.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:21:08.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:21:08.640 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:21:08.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836485 2026-02-13T02:21:08.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836485 2026-02-13T02:21:08.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485' 2026-02-13T02:21:08.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:21:08.713 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:21:08.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705670 2026-02-13T02:21:08.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705670 2026-02-13T02:21:08.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-38654705670' 2026-02-13T02:21:08.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:21:08.786 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:21:08.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574852 2026-02-13T02:21:08.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574852 2026-02-13T02:21:08.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-38654705670 2-55834574852' 2026-02-13T02:21:08.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:21:08.864 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836485 2026-02-13T02:21:08.864 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:21:08.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:21:08.866 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836485 2026-02-13T02:21:08.866 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:21:08.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836485 2026-02-13T02:21:08.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-13T02:21:08.868 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836485 2026-02-13T02:21:08.868 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:21:09.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836485 2026-02-13T02:21:09.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:21:10.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:21:10.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:21:10.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836485 2026-02-13T02:21:10.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:21:11.206 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:21:11.207 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:21:11.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836485 2026-02-13T02:21:11.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:21:11.371 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705670 2026-02-13T02:21:11.371 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:21:11.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:21:11.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705670 2026-02-13T02:21:11.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:21:11.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705670 2026-02-13T02:21:11.376 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705670 2026-02-13T02:21:11.376 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705670' 2026-02-13T02:21:11.376 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:21:11.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705670 -lt 38654705670 2026-02-13T02:21:11.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:21:11.546 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574852 2026-02-13T02:21:11.546 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:21:11.547 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:21:11.549 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574852 2026-02-13T02:21:11.549 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:21:11.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574852 2026-02-13T02:21:11.550 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574852 2026-02-13T02:21:11.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574852' 2026-02-13T02:21:11.551 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:21:11.720 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574852 -lt 55834574852 2026-02-13T02:21:11.720 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:21:11.721 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:21:11.721 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:21:11.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:21:11.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:21:11.920 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:21:11.920 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:21:11.920 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:21:11.920 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:21:11.920 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:21:11.920 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:21:12.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:21:12.081 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:21:12.081 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:21:12.081 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:21:12.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:21:12.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:21:12.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:21:12.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2536: create_ec_pool: return 0 2026-02-13T02:21:12.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:990: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ0 2026-02-13T02:21:12.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:21:12.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-13T02:21:12.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ0 2026-02-13T02:21:12.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:21:12.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:21:12.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:21:12.483 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:21:12.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:21:12.688 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:21:12.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:21:12.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:21:12.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ0 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:21:12.727 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:991: list_missing_erasure_coded: get_osds ecpool MOBJ0 2026-02-13T02:21:12.727 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2026-02-13T02:21:12.727 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=MOBJ0 2026-02-13T02:21:12.728 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool MOBJ0 2026-02-13T02:21:12.728 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-13T02:21:12.892 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=2 2026-02-13T02:21:12.892 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:21:12.892 INFO:tasks.workunit.client.0.trial112.stderr:0' 2026-02-13T02:21:12.893 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 2 1 0 2026-02-13T02:21:12.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:991: list_missing_erasure_coded: osds0=('2' '1' '0') 2026-02-13T02:21:12.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:991: list_missing_erasure_coded: local -a osds0 2026-02-13T02:21:12.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:994: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ1 2026-02-13T02:21:12.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:21:12.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-13T02:21:12.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ1 2026-02-13T02:21:12.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:21:12.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:21:12.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:21:13.098 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:21:13.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:21:13.301 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:21:13.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:21:13.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:21:13.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ1 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:21:13.338 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:995: list_missing_erasure_coded: get_osds ecpool MOBJ1 2026-02-13T02:21:13.338 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2026-02-13T02:21:13.338 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=MOBJ1 2026-02-13T02:21:13.339 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool MOBJ1 2026-02-13T02:21:13.339 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-13T02:21:13.500 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=2 2026-02-13T02:21:13.501 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:21:13.501 INFO:tasks.workunit.client.0.trial112.stderr:0' 2026-02-13T02:21:13.501 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 2 1 0 2026-02-13T02:21:13.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:995: list_missing_erasure_coded: osds1=('2' '1' '0') 2026-02-13T02:21:13.502 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:995: list_missing_erasure_coded: local -a osds1 2026-02-13T02:21:13.502 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:998: list_missing_erasure_coded: seq 0 2 2026-02-13T02:21:13.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:998: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-13T02:21:13.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:999: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T02:21:13.503 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:21:13.503 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:21:13.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:21:13.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:21:13.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:21:13.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:21:13.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:998: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-13T02:21:13.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:999: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:21:13.814 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:21:13.815 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:21:13.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:21:13.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:21:13.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:21:14.125 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:21:14.125 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:998: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-13T02:21:14.125 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:999: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.2 2026-02-13T02:21:14.126 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:21:14.126 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:21:14.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:21:14.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:21:14.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:21:14.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:21:14.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1002: list_missing_erasure_coded: id=2 2026-02-13T02:21:14.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1003: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 MOBJ0 remove 2026-02-13T02:21:14.613 INFO:tasks.workunit.client.0.trial112.stdout:remove 0#2:a9030505:::MOBJ0:head# 2026-02-13T02:21:14.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1005: list_missing_erasure_coded: id=1 2026-02-13T02:21:14.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ0 remove 2026-02-13T02:21:14.880 INFO:tasks.workunit.client.0.trial112.stdout:remove 1#2:a9030505:::MOBJ0:head# 2026-02-13T02:21:14.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1009: list_missing_erasure_coded: id=1 2026-02-13T02:21:14.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ1 remove 2026-02-13T02:21:15.147 INFO:tasks.workunit.client.0.trial112.stdout:remove 1#2:8dd16d55:::MOBJ1:head# 2026-02-13T02:21:15.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1012: list_missing_erasure_coded: id=0 2026-02-13T02:21:15.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1013: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 MOBJ1 remove 2026-02-13T02:21:15.425 INFO:tasks.workunit.client.0.trial112.stdout:remove 2#2:8dd16d55:::MOBJ1:head# 2026-02-13T02:21:15.517 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: list_missing_erasure_coded: seq 0 2 2026-02-13T02:21:15.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-13T02:21:15.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1017: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 0 2026-02-13T02:21:15.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:21:15.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:21:15.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:21:15.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:21:15.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:21:15.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:21:15.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:21:15.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:21:15.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:21:15.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:21:15.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:21:15.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:21:15.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:21:15.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:21:15.522 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:21:15.522 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:21:15.522 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:21:15.522 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:21:15.522 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:21:15.522 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:21:15.524 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:21:15.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:21:15.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:21:15.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:21:15.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:21:15.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:21:15.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:21:15.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:21:15.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:21:15.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:21:15.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:21:15.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:21:15.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:21:15.525 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T02:21:15.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:21:15.525 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T02:21:15.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:21:15.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:21:15.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:21:15.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:21:15.546 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:15.531+0000 7fb19cfdfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:21:15.548 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:15.531+0000 7fb19cfdfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:21:15.549 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:15.535+0000 7fb19cfdfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:21:15.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:21:15.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:21:15.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:21:15.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:21:15.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:21:15.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:21:15.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:21:15.688 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:21:15.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:21:15.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:21:15.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:21:15.906 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:15.891+0000 7fb19cfdfa40 -1 Falling back to public interface 2026-02-13T02:21:16.540 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:16.523+0000 7fb19cfdfa40 -1 osd.0 32 log_to_monitors true 2026-02-13T02:21:16.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:21:16.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:21:16.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:21:16.846 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:21:16.847 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:21:16.847 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:21:17.007 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 39 up_thru 39 down_at 33 last_clean_interval [5,32) [v2:127.0.0.1:6800/2438083343,v1:127.0.0.1:6801/2438083343] [v2:127.0.0.1:6802/2438083343,v1:127.0.0.1:6803/2438083343] exists,up bed6d058-d5fb-4a47-861a-bc1e403ce73b 2026-02-13T02:21:17.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:21:17.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:21:17.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:21:17.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-13T02:21:17.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1017: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 1 2026-02-13T02:21:17.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:21:17.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:21:17.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:21:17.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:21:17.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:21:17.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:21:17.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:21:17.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:21:17.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:21:17.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:21:17.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:21:17.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:21:17.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:21:17.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:21:17.008 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:21:17.009 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:21:17.009 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:21:17.009 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:21:17.009 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:21:17.009 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:21:17.009 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:21:17.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:21:17.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:21:17.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:21:17.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:21:17.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:21:17.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:21:17.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:21:17.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:21:17.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:21:17.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:21:17.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:21:17.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:21:17.012 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:21:17.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:21:17.012 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:21:17.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:21:17.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:21:17.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:21:17.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:21:17.025 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:17.011+0000 7f86d868ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:21:17.026 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:17.011+0000 7f86d868ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:21:17.027 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:17.011+0000 7f86d868ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:21:17.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:21:17.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:21:17.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:21:17.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:21:17.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:21:17.175 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:21:17.175 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:21:17.175 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:21:17.175 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:21:17.175 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:21:17.332 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:21:17.410 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:17.395+0000 7f86d868ea40 -1 Falling back to public interface 2026-02-13T02:21:18.031 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:18.015+0000 7f86d868ea40 -1 osd.1 34 log_to_monitors true 2026-02-13T02:21:18.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:21:18.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:21:18.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:21:18.334 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:21:18.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:21:18.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:21:18.498 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 35 last_clean_interval [9,34) [v2:127.0.0.1:6810/799079379,v1:127.0.0.1:6811/799079379] [v2:127.0.0.1:6812/799079379,v1:127.0.0.1:6813/799079379] exists,up 8984e199-f50d-43c5-98e5-cae80770f4fb 2026-02-13T02:21:18.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:21:18.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:21:18.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:21:18.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-13T02:21:18.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1017: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 2 2026-02-13T02:21:18.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:21:18.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:21:18.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-13T02:21:18.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:21:18.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2026-02-13T02:21:18.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:21:18.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:21:18.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:21:18.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:21:18.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-13T02:21:18.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-13T02:21:18.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:21:18.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:21:18.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:21:18.500 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:21:18.500 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:21:18.500 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:21:18.500 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:21:18.501 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:21:18.501 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:21:18.501 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:21:18.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:21:18.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:21:18.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:21:18.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:21:18.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:21:18.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:21:18.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:21:18.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:21:18.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:21:18.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:21:18.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2026-02-13T02:21:18.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-13T02:21:18.504 INFO:tasks.workunit.client.0.trial112.stderr:start osd.2 2026-02-13T02:21:18.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:21:18.504 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2026-02-13T02:21:18.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-13T02:21:18.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:21:18.505 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:21:18.505 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:21:18.515 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:18.499+0000 7f3b7e551a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:21:18.516 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:18.503+0000 7f3b7e551a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:21:18.517 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:18.503+0000 7f3b7e551a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:21:18.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-13T02:21:18.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:21:18.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T02:21:18.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:21:18.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:21:18.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:21:18.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:21:18.669 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:21:18.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:21:18.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:21:18.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:21:18.910 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:18.895+0000 7f3b7e551a40 -1 Falling back to public interface 2026-02-13T02:21:19.531 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:19.515+0000 7f3b7e551a40 -1 osd.2 36 log_to_monitors true 2026-02-13T02:21:19.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:21:19.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:21:19.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:21:19.831 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:21:19.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:21:19.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:21:19.988 INFO:tasks.workunit.client.0.trial112.stderr:osd.2 up in weight 1 up_from 48 up_thru 48 down_at 37 last_clean_interval [13,36) [v2:127.0.0.1:6818/4131365090,v1:127.0.0.1:6819/4131365090] [v2:127.0.0.1:6820/4131365090,v1:127.0.0.1:6821/4131365090] exists,up 872361e8-6578-41a3-a84e-79a9afcabd6a 2026-02-13T02:21:19.988 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:21:19.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:21:19.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:21:19.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: list_missing_erasure_coded: create_rbd_pool 2026-02-13T02:21:19.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-13T02:21:20.194 INFO:tasks.workunit.client.0.trial112.stderr:pool 'rbd' removed 2026-02-13T02:21:20.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-13T02:21:20.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-13T02:21:20.397 INFO:tasks.workunit.client.0.trial112.stderr:pool 'rbd' created 2026-02-13T02:21:20.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:21:21.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-13T02:21:21.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: list_missing_erasure_coded: wait_for_clean 2026-02-13T02:21:21.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:21:21.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:21:21.692 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:21:21.692 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:21:21.693 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:21:21.693 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:21:21.693 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:21:21.694 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:21:21.694 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:21:21.816 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:21:21.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:21:21.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:21:21.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:21:21.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:21:21.817 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:21:21.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:21:21.976 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:21:21.976 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:21:21.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:21:21.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:21:21.977 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:21:22.052 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=167503724547 2026-02-13T02:21:22.052 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 167503724547 2026-02-13T02:21:22.052 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-167503724547' 2026-02-13T02:21:22.052 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:21:22.053 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:21:22.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=184683593730 2026-02-13T02:21:22.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 184683593730 2026-02-13T02:21:22.128 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-167503724547 1-184683593730' 2026-02-13T02:21:22.128 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:21:22.128 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:21:22.202 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=206158430210 2026-02-13T02:21:22.202 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 206158430210 2026-02-13T02:21:22.202 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-167503724547 1-184683593730 2-206158430210' 2026-02-13T02:21:22.202 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:21:22.203 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-167503724547 2026-02-13T02:21:22.203 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:21:22.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:21:22.206 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-167503724547 2026-02-13T02:21:22.206 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:21:22.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=167503724547 2026-02-13T02:21:22.208 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 167503724547 2026-02-13T02:21:22.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 167503724547' 2026-02-13T02:21:22.208 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:21:22.369 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 167503724547 2026-02-13T02:21:22.369 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:21:23.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:21:23.371 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:21:23.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 167503724547 -lt 167503724547 2026-02-13T02:21:23.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:21:23.533 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-184683593730 2026-02-13T02:21:23.533 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:21:23.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:21:23.536 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-184683593730 2026-02-13T02:21:23.536 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:21:23.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=184683593730 2026-02-13T02:21:23.538 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 184683593730 2026-02-13T02:21:23.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 184683593730' 2026-02-13T02:21:23.539 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:21:23.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 184683593730 -lt 184683593730 2026-02-13T02:21:23.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:21:23.712 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-206158430210 2026-02-13T02:21:23.712 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:21:23.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:21:23.714 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-206158430210 2026-02-13T02:21:23.714 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:21:23.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=206158430210 2026-02-13T02:21:23.716 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 206158430210 2026-02-13T02:21:23.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 206158430210' 2026-02-13T02:21:23.717 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:21:23.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 206158430210 -lt 206158430210 2026-02-13T02:21:23.885 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:21:23.885 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:21:23.886 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:21:24.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:21:24.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:21:24.084 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:21:24.084 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:21:24.085 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:21:24.085 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:21:24.085 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:21:24.085 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:21:24.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:21:24.247 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:21:24.247 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:21:24.247 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:21:24.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:21:24.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:21:24.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:21:24.446 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: list_missing_erasure_coded: get_pg ecpool MOBJ0 2026-02-13T02:21:24.447 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-13T02:21:24.447 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=MOBJ0 2026-02-13T02:21:24.447 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool MOBJ0 2026-02-13T02:21:24.447 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T02:21:24.615 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: list_missing_erasure_coded: local pg=2.0 2026-02-13T02:21:24.615 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: list_missing_erasure_coded: repair 2.0 2026-02-13T02:21:24.615 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-13T02:21:24.615 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-13T02:21:24.615 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:21:24.615 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:21:24.615 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:21:24.615 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:21:24.775 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-13T02:21:07.100265+0000 2026-02-13T02:21:24.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-13T02:21:24.927 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 2.0s0 on osd.2 to repair 2026-02-13T02:21:24.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-13T02:21:07.100265+0000 2026-02-13T02:21:24.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-13T02:21:24.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:21:07.100265+0000 2026-02-13T02:21:24.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:21:24.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:21:24.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:21:24.936 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:21:24.936 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:21:24.936 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:21:24.937 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:21:24.937 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:21:25.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:21:07.100265+0000 '>' 2026-02-13T02:21:07.100265+0000 2026-02-13T02:21:25.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:21:26.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:21:26.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:21:26.103 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:21:26.103 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:21:26.103 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:21:26.104 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:21:26.104 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:21:26.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:21:07.100265+0000 '>' 2026-02-13T02:21:07.100265+0000 2026-02-13T02:21:26.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:21:27.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:21:27.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:21:27.263 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:21:27.263 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:21:27.263 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:21:27.264 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:21:27.264 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:21:27.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:21:07.100265+0000 '>' 2026-02-13T02:21:07.100265+0000 2026-02-13T02:21:27.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:21:28.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:21:28.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:21:28.424 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:21:28.424 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:21:28.424 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:21:28.425 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:21:28.425 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:21:28.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:21:07.100265+0000 '>' 2026-02-13T02:21:07.100265+0000 2026-02-13T02:21:28.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:21:29.584 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:21:29.584 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:21:29.585 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:21:29.585 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:21:29.585 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:21:29.585 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:21:29.585 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:21:29.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:21:07.100265+0000 '>' 2026-02-13T02:21:07.100265+0000 2026-02-13T02:21:29.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:21:30.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:21:30.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:21:30.743 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-13T02:21:30.743 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-13T02:21:30.743 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:21:30.744 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:21:30.744 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-13T02:21:30.910 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:21:25.411837+0000 '>' 2026-02-13T02:21:07.100265+0000 2026-02-13T02:21:30.910 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:21:30.910 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: list_missing_erasure_coded: seq 0 120 2026-02-13T02:21:30.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: list_missing_erasure_coded: for i in $(seq 0 120) 2026-02-13T02:21:30.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: list_missing_erasure_coded: '[' 0 -lt 60 ']' 2026-02-13T02:21:30.913 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1031: list_missing_erasure_coded: ceph pg 2.0 list_unfound 2026-02-13T02:21:30.913 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1031: list_missing_erasure_coded: egrep 'MOBJ0|MOBJ1' 2026-02-13T02:21:30.913 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1031: list_missing_erasure_coded: wc -l 2026-02-13T02:21:30.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1031: list_missing_erasure_coded: matches=2 2026-02-13T02:21:30.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1032: list_missing_erasure_coded: '[' 2 -eq 2 ']' 2026-02-13T02:21:30.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1032: list_missing_erasure_coded: break 2026-02-13T02:21:30.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-13T02:21:30.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:21:30.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:21:30.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:21:30.990 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:21:30.990 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:21:30.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:21:30.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:21:30.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:21:31.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:21:31.303 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:21:31.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:21:31.306 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:21:31.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:21:31.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:21:31.308 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:21:31.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:21:31.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:21:31.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:21:31.310 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:21:31.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:21:31.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:21:31.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:21:31.328 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:21:31.328 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:21:31.328 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:21:31.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:21:31.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:21:31.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:21:31.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-13T02:21:31.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-13T02:21:31.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-13T02:21:31.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-13T02:21:31.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:21:31.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:21:31.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:21:31.330 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:21:31.330 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:21:31.331 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:21:31.331 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:21:31.331 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:21:31.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:21:31.334 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:21:31.336 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:21:31.336 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:21:31.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:21:31.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:21:31.338 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:21:31.339 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:21:31.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:21:31.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:21:31.340 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:21:31.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:21:31.343 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:21:31.343 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:21:31.344 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:21:31.344 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:21:31.344 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:21:31.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:21:31.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:21:31.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:21:31.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-13T02:21:31.347 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T02:21:31.347 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:21:31.348 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:21:31.348 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.35207 2026-02-13T02:21:31.349 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T02:21:31.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-13T02:21:31.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T02:21:31.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-13T02:21:31.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_periodic_scrub_replicated td/osd-scrub-repair 2026-02-13T02:21:31.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5926: TEST_periodic_scrub_replicated: local dir=td/osd-scrub-repair 2026-02-13T02:21:31.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5927: TEST_periodic_scrub_replicated: local poolname=psr_pool 2026-02-13T02:21:31.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5928: TEST_periodic_scrub_replicated: local objname=POBJ 2026-02-13T02:21:31.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5929: TEST_periodic_scrub_replicated: return 0 2026-02-13T02:21:31.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-13T02:21:31.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:21:31.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:21:31.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:21:31.352 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:21:31.352 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:21:31.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:21:31.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:21:31.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:21:31.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:21:31.355 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:21:31.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:21:31.357 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:21:31.358 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:21:31.358 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:21:31.358 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:21:31.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:21:31.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:21:31.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:21:31.361 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:21:31.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:21:31.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:21:31.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:21:31.364 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:21:31.364 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:21:31.364 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:21:31.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:21:31.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:21:31.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:21:31.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-13T02:21:31.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-13T02:21:31.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-13T02:21:31.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-13T02:21:31.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:21:31.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:21:31.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:21:31.367 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:21:31.368 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:21:31.368 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:21:31.368 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:21:31.368 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:21:31.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:21:31.370 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:21:31.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:21:31.372 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:21:31.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:21:31.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:21:31.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:21:31.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:21:31.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:21:31.376 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:21:31.376 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:21:31.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:21:31.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:21:31.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:21:31.380 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:21:31.380 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:21:31.381 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:21:31.381 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:21:31.382 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:21:31.382 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:21:31.382 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-13T02:21:31.384 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T02:21:31.384 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:21:31.384 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:21:31.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.35207 2026-02-13T02:21:31.386 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T02:21:31.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-13T02:21:31.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T02:21:31.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-13T02:21:31.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_repair_stats td/osd-scrub-repair 2026-02-13T02:21:31.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:713: TEST_repair_stats: local dir=td/osd-scrub-repair 2026-02-13T02:21:31.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:714: TEST_repair_stats: local poolname=testpool 2026-02-13T02:21:31.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: local OSDS=2 2026-02-13T02:21:31.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: TEST_repair_stats: local OBJS=30 2026-02-13T02:21:31.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: TEST_repair_stats: local REPAIRS=20 2026-02-13T02:21:31.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:721: TEST_repair_stats: run_mon td/osd-scrub-repair a 2026-02-13T02:21:31.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-13T02:21:31.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-13T02:21:31.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-13T02:21:31.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-13T02:21:31.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-13T02:21:31.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-13T02:21:31.410 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-13T02:21:31.410 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:21:31.410 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:21:31.410 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:21:31.411 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:21:31.411 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:21:31.411 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:21:31.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-13T02:21:31.439 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-13T02:21:31.440 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-13T02:21:31.440 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:21:31.440 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:21:31.440 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-13T02:21:31.441 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-13T02:21:31.442 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:21:31.442 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:21:31.442 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:21:31.442 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:21:31.442 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:21:31.442 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:21:31.443 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:21:31.443 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:21:31.443 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get fsid 2026-02-13T02:21:31.500 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-13T02:21:31.500 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:21:31.500 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:21:31.500 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-13T02:21:31.500 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-13T02:21:31.501 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:21:31.501 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:21:31.501 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:21:31.501 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:21:31.502 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:21:31.502 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:21:31.503 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:21:31.503 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:21:31.503 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get mon_host 2026-02-13T02:21:31.558 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:722: TEST_repair_stats: run_mgr td/osd-scrub-repair x 2026-02-13T02:21:31.558 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-13T02:21:31.558 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-13T02:21:31.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-13T02:21:31.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-13T02:21:31.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-13T02:21:31.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-13T02:21:31.655 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-13T02:21:31.655 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:21:31.655 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:21:31.655 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:21:31.655 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:21:31.655 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:21:31.656 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:21:31.656 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:21:31.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:21:31.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:724: TEST_repair_stats: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-13T02:21:31.678 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:725: TEST_repair_stats: expr 2 - 1 2026-02-13T02:21:31.680 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:725: TEST_repair_stats: seq 0 1 2026-02-13T02:21:31.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:725: TEST_repair_stats: for id in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:21:31.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:726: TEST_repair_stats: run_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T02:21:31.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:21:31.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:21:31.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-13T02:21:31.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:21:31.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:21:31.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:21:31.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:21:31.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:21:31.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:21:31.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:21:31.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:21:31.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:21:31.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:21:31.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:21:31.685 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:21:31.685 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:21:31.685 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:21:31.685 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:21:31.685 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:21:31.686 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:21:31.686 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:21:31.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:21:31.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:21:31.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:21:31.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:21:31.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:21:31.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:21:31.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:21:31.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:21:31.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:21:31.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:21:31.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:21:31.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-13T02:21:31.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:21:31.687 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:21:31.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=948b2763-7fb8-44d6-a0eb-ef4e838438e5 2026-02-13T02:21:31.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 948b2763-7fb8-44d6-a0eb-ef4e838438e5' 2026-02-13T02:21:31.688 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 948b2763-7fb8-44d6-a0eb-ef4e838438e5 2026-02-13T02:21:31.688 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:21:31.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCrio5p4kbSKBAASj8rHcLRVvtP6WFTL4m7KQ== 2026-02-13T02:21:31.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCrio5p4kbSKBAASj8rHcLRVvtP6WFTL4m7KQ=="}' 2026-02-13T02:21:31.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 948b2763-7fb8-44d6-a0eb-ef4e838438e5 -i td/osd-scrub-repair/0/new.json 2026-02-13T02:21:31.786 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:21:31.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-13T02:21:31.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCrio5p4kbSKBAASj8rHcLRVvtP6WFTL4m7KQ== --osd-uuid 948b2763-7fb8-44d6-a0eb-ef4e838438e5 2026-02-13T02:21:31.812 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:31.798+0000 7fbd7bbf2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:21:31.813 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:31.802+0000 7fbd7bbf2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:21:31.814 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:31.802+0000 7fbd7bbf2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:21:31.814 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:31.802+0000 7fbd7bbf2a40 -1 bdev(0x5561710aec00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-13T02:21:31.815 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:31.802+0000 7fbd7bbf2a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-13T02:21:32.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-13T02:21:32.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:21:32.550 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T02:21:32.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-13T02:21:32.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:21:32.657 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T02:21:32.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-13T02:21:32.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T02:21:32.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:21:32.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:21:32.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:21:32.683 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:32.670+0000 7f507c1d1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:21:32.685 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:32.674+0000 7f507c1d1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:21:32.687 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:32.674+0000 7f507c1d1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:21:32.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-13T02:21:32.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:21:32.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:21:32.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:21:32.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:21:32.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:21:32.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:21:32.755 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:21:32.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:21:32.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:21:32.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:21:33.166 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:33.154+0000 7f507c1d1a40 -1 Falling back to public interface 2026-02-13T02:21:33.848 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:21:33.848 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:21:33.848 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:21:33.848 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:21:33.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:21:33.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:21:33.944 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:21:34.113 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:34.098+0000 7f507c1d1a40 -1 osd.0 0 log_to_monitors true 2026-02-13T02:21:34.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:21:34.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:21:34.947 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:21:34.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:21:34.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:21:34.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:21:35.111 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3262145067,v1:127.0.0.1:6801/3262145067] [v2:127.0.0.1:6802/3262145067,v1:127.0.0.1:6803/3262145067] exists,up 948b2763-7fb8-44d6-a0eb-ef4e838438e5 2026-02-13T02:21:35.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:21:35.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:21:35.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:21:35.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:725: TEST_repair_stats: for id in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:21:35.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:726: TEST_repair_stats: run_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T02:21:35.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:21:35.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:21:35.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-13T02:21:35.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:21:35.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:21:35.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:21:35.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:21:35.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:21:35.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:21:35.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:21:35.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:21:35.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:21:35.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:21:35.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:21:35.117 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:21:35.117 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:21:35.117 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:21:35.117 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:21:35.117 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:21:35.117 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:21:35.117 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:21:35.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:21:35.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:21:35.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:21:35.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:21:35.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:21:35.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:21:35.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:21:35.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:21:35.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:21:35.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:21:35.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:21:35.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-13T02:21:35.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:21:35.119 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:21:35.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=955a4103-83c9-4e9e-b3e3-66afb731a19c 2026-02-13T02:21:35.120 INFO:tasks.workunit.client.0.trial112.stdout:add osd1 955a4103-83c9-4e9e-b3e3-66afb731a19c 2026-02-13T02:21:35.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 955a4103-83c9-4e9e-b3e3-66afb731a19c' 2026-02-13T02:21:35.120 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:21:35.129 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCvio5prLMNBxAAvDkHFWCIEhApQ70fuaFuzQ== 2026-02-13T02:21:35.129 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCvio5prLMNBxAAvDkHFWCIEhApQ70fuaFuzQ=="}' 2026-02-13T02:21:35.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 955a4103-83c9-4e9e-b3e3-66afb731a19c -i td/osd-scrub-repair/1/new.json 2026-02-13T02:21:35.291 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:21:35.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-13T02:21:35.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCvio5prLMNBxAAvDkHFWCIEhApQ70fuaFuzQ== --osd-uuid 955a4103-83c9-4e9e-b3e3-66afb731a19c 2026-02-13T02:21:35.326 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:35.314+0000 7efd8d6b2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:21:35.328 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:35.314+0000 7efd8d6b2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:21:35.329 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:35.318+0000 7efd8d6b2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:21:35.330 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:35.318+0000 7efd8d6b2a40 -1 bdev(0x561011de2c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-13T02:21:35.330 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:35.318+0000 7efd8d6b2a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-13T02:21:35.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-13T02:21:35.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:21:35.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-13T02:21:35.997 INFO:tasks.workunit.client.0.trial112.stdout:adding osd1 key to auth repository 2026-02-13T02:21:35.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:21:36.207 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T02:21:36.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-13T02:21:36.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T02:21:36.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:21:36.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:21:36.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:21:36.231 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:36.218+0000 7f34eb307a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:21:36.233 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:36.222+0000 7f34eb307a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:21:36.234 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:36.222+0000 7f34eb307a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:21:36.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-13T02:21:36.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:21:36.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:21:36.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:21:36.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:21:36.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:21:36.366 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:21:36.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:21:36.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:21:36.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:21:36.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:21:36.578 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:36.566+0000 7f34eb307a40 -1 Falling back to public interface 2026-02-13T02:21:37.202 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:37.190+0000 7f34eb307a40 -1 osd.1 0 log_to_monitors true 2026-02-13T02:21:37.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:21:37.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:21:37.528 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:21:37.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:21:37.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:21:37.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:21:37.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:21:38.694 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:21:38.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:21:38.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:21:38.695 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:21:38.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:21:38.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:21:38.858 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2933945492,v1:127.0.0.1:6811/2933945492] [v2:127.0.0.1:6812/2933945492,v1:127.0.0.1:6813/2933945492] exists,up 955a4103-83c9-4e9e-b3e3-66afb731a19c 2026-02-13T02:21:38.859 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:21:38.859 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:21:38.859 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:21:38.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:729: TEST_repair_stats: create_pool testpool 1 1 2026-02-13T02:21:38.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create testpool 1 1 2026-02-13T02:21:39.069 INFO:tasks.workunit.client.0.trial112.stderr:pool 'testpool' created 2026-02-13T02:21:39.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:21:40.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:730: TEST_repair_stats: ceph osd pool set testpool size 2 2026-02-13T02:21:40.289 INFO:tasks.workunit.client.0.trial112.stderr:set pool 1 size to 2 2026-02-13T02:21:40.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:731: TEST_repair_stats: wait_for_clean 2026-02-13T02:21:40.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:21:40.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:21:40.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:21:40.301 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:21:40.301 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:21:40.301 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:21:40.302 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:21:40.302 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:21:40.302 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:21:40.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:21:40.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:21:40.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:21:40.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:21:40.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:21:40.432 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:21:40.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:21:40.592 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:21:40.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:21:40.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:21:40.593 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:21:40.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-13T02:21:40.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-13T02:21:40.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-13T02:21:40.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:21:40.667 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:21:40.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705666 2026-02-13T02:21:40.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705666 2026-02-13T02:21:40.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705666' 2026-02-13T02:21:40.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:21:40.742 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-13T02:21:40.742 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:21:40.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:21:40.744 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-13T02:21:40.744 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:21:40.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-13T02:21:40.746 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836483 2026-02-13T02:21:40.747 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-13T02:21:40.747 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:21:40.915 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-13T02:21:40.915 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:21:41.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:21:41.917 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:21:42.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-13T02:21:42.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:21:42.078 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705666 2026-02-13T02:21:42.078 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:21:42.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:21:42.081 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705666 2026-02-13T02:21:42.081 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:21:42.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705666 2026-02-13T02:21:42.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705666' 2026-02-13T02:21:42.083 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705666 2026-02-13T02:21:42.083 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:21:42.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705666 -lt 38654705666 2026-02-13T02:21:42.239 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:21:42.239 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:21:42.239 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:21:42.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:21:42.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:21:42.432 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:21:42.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:21:42.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:21:42.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:21:42.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:21:42.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:21:42.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:21:42.596 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:21:42.596 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:21:42.597 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:21:42.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:21:42.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:21:42.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:21:42.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:734: TEST_repair_stats: local payload=ABCDEF 2026-02-13T02:21:42.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:735: TEST_repair_stats: echo ABCDEF 2026-02-13T02:21:42.793 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: seq 1 30 2026-02-13T02:21:42.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-13T02:21:42.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:21:42.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-13T02:21:42.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:21:42.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-13T02:21:42.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:21:42.866 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-13T02:21:42.866 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:21:42.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-13T02:21:42.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:21:42.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-13T02:21:42.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:21:42.944 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-13T02:21:42.944 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:21:42.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-13T02:21:42.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:21:42.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-13T02:21:42.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:21:43.021 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-13T02:21:43.021 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:21:43.047 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-13T02:21:43.047 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj11 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:21:43.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-13T02:21:43.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj12 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:21:43.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-13T02:21:43.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj13 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:21:43.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-13T02:21:43.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj14 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:21:43.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-13T02:21:43.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj15 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:21:43.167 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-13T02:21:43.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj16 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:21:43.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-13T02:21:43.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj17 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:21:43.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-13T02:21:43.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj18 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:21:43.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-13T02:21:43.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj19 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:21:43.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-13T02:21:43.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj20 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:21:43.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-13T02:21:43.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj21 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:21:43.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-13T02:21:43.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj22 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:21:43.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-13T02:21:43.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj23 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:21:43.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-13T02:21:43.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj24 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:21:43.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-13T02:21:43.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj25 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:21:43.414 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-13T02:21:43.414 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj26 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:21:43.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-13T02:21:43.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj27 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:21:43.465 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-13T02:21:43.465 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj28 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:21:43.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-13T02:21:43.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj29 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:21:43.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-13T02:21:43.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj30 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:21:43.543 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:743: TEST_repair_stats: get_not_primary testpool obj1 2026-02-13T02:21:43.543 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-13T02:21:43.543 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=obj1 2026-02-13T02:21:43.544 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool obj1 2026-02-13T02:21:43.544 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-13T02:21:43.544 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-13T02:21:43.545 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2026-02-13T02:21:43.545 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T02:21:43.706 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-13T02:21:43.707 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool obj1 2026-02-13T02:21:43.707 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-13T02:21:43.869 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:743: TEST_repair_stats: local other=0 2026-02-13T02:21:43.870 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:744: TEST_repair_stats: get_pg testpool obj1 2026-02-13T02:21:43.871 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2026-02-13T02:21:43.871 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=obj1 2026-02-13T02:21:43.871 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool obj1 2026-02-13T02:21:43.871 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T02:21:44.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:744: TEST_repair_stats: local pgid=1.0 2026-02-13T02:21:44.035 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:745: TEST_repair_stats: get_primary testpool obj1 2026-02-13T02:21:44.036 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-13T02:21:44.036 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-13T02:21:44.036 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2026-02-13T02:21:44.036 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T02:21:44.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:745: TEST_repair_stats: local primary=1 2026-02-13T02:21:44.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T02:21:44.200 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:21:44.200 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:21:44.201 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:21:44.201 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:21:44.201 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:21:44.511 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:21:44.511 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:748: TEST_repair_stats: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:21:44.511 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:21:44.511 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:21:44.512 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:21:44.512 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:21:44.512 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:21:44.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:21:44.823 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: seq 1 20 2026-02-13T02:21:44.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-13T02:21:44.825 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 1 % 2 2026-02-13T02:21:44.827 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=1 2026-02-13T02:21:44.827 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 remove 2026-02-13T02:21:44.827 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:21:44.827 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:21:44.827 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:21:44.827 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:21:44.827 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:21:44.827 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 remove 2026-02-13T02:21:44.993 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:ff7b1f36:::obj1:head# 2026-02-13T02:21:45.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-13T02:21:45.117 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 2 % 2 2026-02-13T02:21:45.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=0 2026-02-13T02:21:45.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2026-02-13T02:21:45.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:21:45.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:21:45.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:21:45.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:21:45.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:21:45.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2026-02-13T02:21:45.297 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:104778fc:::obj2:head# 2026-02-13T02:21:45.401 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-13T02:21:45.401 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 3 % 2 2026-02-13T02:21:45.403 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=1 2026-02-13T02:21:45.403 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj3 remove 2026-02-13T02:21:45.403 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:21:45.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:21:45.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:21:45.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:21:45.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:21:45.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj3 remove 2026-02-13T02:21:45.560 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:8dd16f86:::obj3:head# 2026-02-13T02:21:45.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-13T02:21:45.677 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 4 % 2 2026-02-13T02:21:45.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=0 2026-02-13T02:21:45.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj4 remove 2026-02-13T02:21:45.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:21:45.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:21:45.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:21:45.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:21:45.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:21:45.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj4 remove 2026-02-13T02:21:45.844 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:0ee9ae15:::obj4:head# 2026-02-13T02:21:45.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-13T02:21:45.949 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 5 % 2 2026-02-13T02:21:45.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=1 2026-02-13T02:21:45.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj5 remove 2026-02-13T02:21:45.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:21:45.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:21:45.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:21:45.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:21:45.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:21:45.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj5 remove 2026-02-13T02:21:46.118 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:c52c9666:::obj5:head# 2026-02-13T02:21:46.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-13T02:21:46.225 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 6 % 2 2026-02-13T02:21:46.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=0 2026-02-13T02:21:46.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj6 remove 2026-02-13T02:21:46.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:21:46.228 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:21:46.228 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:21:46.228 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:21:46.228 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:21:46.228 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj6 remove 2026-02-13T02:21:46.415 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:233a42c1:::obj6:head# 2026-02-13T02:21:46.512 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-13T02:21:46.513 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 7 % 2 2026-02-13T02:21:46.515 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=1 2026-02-13T02:21:46.515 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj7 remove 2026-02-13T02:21:46.515 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:21:46.515 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:21:46.515 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:21:46.515 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:21:46.515 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:21:46.515 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj7 remove 2026-02-13T02:21:46.679 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:9a09113a:::obj7:head# 2026-02-13T02:21:46.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-13T02:21:46.785 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 8 % 2 2026-02-13T02:21:46.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=0 2026-02-13T02:21:46.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj8 remove 2026-02-13T02:21:46.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:21:46.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:21:46.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:21:46.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:21:46.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:21:46.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj8 remove 2026-02-13T02:21:46.948 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:a557efb1:::obj8:head# 2026-02-13T02:21:47.065 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-13T02:21:47.065 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 9 % 2 2026-02-13T02:21:47.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=1 2026-02-13T02:21:47.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj9 remove 2026-02-13T02:21:47.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:21:47.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:21:47.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:21:47.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:21:47.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:21:47.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj9 remove 2026-02-13T02:21:47.236 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:fe351e27:::obj9:head# 2026-02-13T02:21:47.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-13T02:21:47.337 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 10 % 2 2026-02-13T02:21:47.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=0 2026-02-13T02:21:47.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj10 remove 2026-02-13T02:21:47.339 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:21:47.339 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:21:47.339 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:21:47.339 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:21:47.339 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:21:47.339 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj10 remove 2026-02-13T02:21:47.514 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:d1337354:::obj10:head# 2026-02-13T02:21:47.620 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-13T02:21:47.621 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 11 % 2 2026-02-13T02:21:47.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=1 2026-02-13T02:21:47.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj11 remove 2026-02-13T02:21:47.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:21:47.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:21:47.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:21:47.624 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:21:47.624 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:21:47.624 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj11 remove 2026-02-13T02:21:47.828 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:eeae2a94:::obj11:head# 2026-02-13T02:21:47.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-13T02:21:47.921 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 12 % 2 2026-02-13T02:21:47.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=0 2026-02-13T02:21:47.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj12 remove 2026-02-13T02:21:47.924 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:21:47.924 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:21:47.924 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:21:47.924 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:21:47.924 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:21:47.924 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj12 remove 2026-02-13T02:21:48.092 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:af5f95cb:::obj12:head# 2026-02-13T02:21:48.201 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-13T02:21:48.201 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 13 % 2 2026-02-13T02:21:48.203 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=1 2026-02-13T02:21:48.203 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj13 remove 2026-02-13T02:21:48.203 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:21:48.203 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:21:48.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:21:48.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:21:48.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:21:48.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj13 remove 2026-02-13T02:21:48.387 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:a61ad63e:::obj13:head# 2026-02-13T02:21:48.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-13T02:21:48.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 14 % 2 2026-02-13T02:21:48.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=0 2026-02-13T02:21:48.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj14 remove 2026-02-13T02:21:48.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:21:48.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:21:48.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:21:48.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:21:48.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:21:48.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj14 remove 2026-02-13T02:21:48.652 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:e0a44829:::obj14:head# 2026-02-13T02:21:48.760 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-13T02:21:48.761 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 15 % 2 2026-02-13T02:21:48.762 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=1 2026-02-13T02:21:48.762 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj15 remove 2026-02-13T02:21:48.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:21:48.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:21:48.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:21:48.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:21:48.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:21:48.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj15 remove 2026-02-13T02:21:48.944 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:ab946124:::obj15:head# 2026-02-13T02:21:49.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-13T02:21:49.033 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 16 % 2 2026-02-13T02:21:49.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=0 2026-02-13T02:21:49.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj16 remove 2026-02-13T02:21:49.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:21:49.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:21:49.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:21:49.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:21:49.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:21:49.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj16 remove 2026-02-13T02:21:49.200 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:461f8b5e:::obj16:head# 2026-02-13T02:21:49.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-13T02:21:49.302 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 17 % 2 2026-02-13T02:21:49.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=1 2026-02-13T02:21:49.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj17 remove 2026-02-13T02:21:49.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:21:49.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:21:49.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:21:49.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:21:49.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:21:49.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj17 remove 2026-02-13T02:21:49.483 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:24fc6e92:::obj17:head# 2026-02-13T02:21:49.580 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-13T02:21:49.581 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 18 % 2 2026-02-13T02:21:49.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=0 2026-02-13T02:21:49.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj18 remove 2026-02-13T02:21:49.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:21:49.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:21:49.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:21:49.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:21:49.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:21:49.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj18 remove 2026-02-13T02:21:49.768 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:b9836a99:::obj18:head# 2026-02-13T02:21:49.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-13T02:21:49.868 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 19 % 2 2026-02-13T02:21:49.869 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=1 2026-02-13T02:21:49.869 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj19 remove 2026-02-13T02:21:49.869 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:21:49.869 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:21:49.869 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:21:49.869 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:21:49.869 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:21:49.869 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj19 remove 2026-02-13T02:21:50.050 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:00b31b6c:::obj19:head# 2026-02-13T02:21:50.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-13T02:21:50.149 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 20 % 2 2026-02-13T02:21:50.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=0 2026-02-13T02:21:50.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj20 remove 2026-02-13T02:21:50.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:21:50.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:21:50.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:21:50.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:21:50.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:21:50.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj20 remove 2026-02-13T02:21:50.328 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:24b5611b:::obj20:head# 2026-02-13T02:21:50.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:755: TEST_repair_stats: activate_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T02:21:50.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:21:50.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:21:50.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:21:50.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:21:50.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:21:50.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:21:50.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:21:50.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:21:50.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:21:50.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:21:50.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:21:50.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:21:50.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:21:50.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:21:50.425 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:21:50.425 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:21:50.425 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:21:50.426 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:21:50.426 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:21:50.426 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:21:50.426 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:21:50.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:21:50.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:21:50.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:21:50.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:21:50.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:21:50.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:21:50.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:21:50.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:21:50.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:21:50.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-13T02:21:50.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:21:50.428 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T02:21:50.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:21:50.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T02:21:50.429 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:21:50.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:21:50.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:21:50.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:21:50.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:21:50.445 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:50.434+0000 7fe1bdae8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:21:50.446 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:50.434+0000 7fe1bdae8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:21:50.447 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:50.434+0000 7fe1bdae8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:21:50.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:21:50.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:21:50.596 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:21:50.596 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:21:50.596 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:21:50.596 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:21:50.596 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:21:50.596 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:21:50.597 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:21:50.597 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:21:50.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:21:50.814 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:50.802+0000 7fe1bdae8a40 -1 Falling back to public interface 2026-02-13T02:21:51.460 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:51.446+0000 7fe1bdae8a40 -1 osd.1 17 log_to_monitors true 2026-02-13T02:21:51.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:21:51.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:21:51.757 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:21:51.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:21:51.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:21:51.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:21:51.915 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 20 up_thru 20 down_at 18 last_clean_interval [9,17) [v2:127.0.0.1:6800/1414889440,v1:127.0.0.1:6801/1414889440] [v2:127.0.0.1:6802/1414889440,v1:127.0.0.1:6803/1414889440] exists,up 955a4103-83c9-4e9e-b3e3-66afb731a19c 2026-02-13T02:21:51.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:21:51.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:21:51.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:21:51.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:756: TEST_repair_stats: activate_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T02:21:51.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:21:51.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:21:51.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:21:51.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:21:51.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:21:51.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:21:51.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:21:51.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:21:51.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:21:51.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:21:51.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:21:51.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:21:51.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:21:51.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:21:51.920 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:21:51.920 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:21:51.920 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:21:51.920 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:21:51.921 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:21:51.921 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:21:51.921 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:21:51.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:21:51.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:21:51.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:21:51.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:21:51.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:21:51.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:21:51.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:21:51.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:21:51.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:21:51.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-13T02:21:51.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:21:51.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:21:51.922 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T02:21:51.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T02:21:51.923 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T02:21:51.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:21:51.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:21:51.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:21:51.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:21:51.936 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:51.922+0000 7fa0a48fca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:21:51.937 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:51.926+0000 7fa0a48fca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:21:51.938 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:51.926+0000 7fa0a48fca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:21:52.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:21:52.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:21:52.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:21:52.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:21:52.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:21:52.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:21:52.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:21:52.088 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:21:52.089 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:21:52.089 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:21:52.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:21:52.322 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:52.310+0000 7fa0a48fca40 -1 Falling back to public interface 2026-02-13T02:21:52.872 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:21:52.858+0000 7fa0a48fca40 -1 osd.0 15 log_to_monitors true 2026-02-13T02:21:53.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:21:53.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:21:53.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:21:53.248 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:21:53.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:21:53.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:21:53.411 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 23 up_thru 0 down_at 16 last_clean_interval [5,15) [v2:127.0.0.1:6810/3396525854,v1:127.0.0.1:6811/3396525854] [v2:127.0.0.1:6812/3396525854,v1:127.0.0.1:6813/3396525854] exists,up 948b2763-7fb8-44d6-a0eb-ef4e838438e5 2026-02-13T02:21:53.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:21:53.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:21:53.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:21:53.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:757: TEST_repair_stats: wait_for_clean 2026-02-13T02:21:53.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:21:53.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:21:53.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:21:53.413 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:21:53.414 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:21:53.414 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:21:53.414 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:21:53.414 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:21:53.414 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:21:53.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:21:53.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:21:53.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:21:53.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:21:53.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:21:53.530 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:21:53.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:21:53.691 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:21:53.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:21:53.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:21:53.691 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:21:53.764 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247810 2026-02-13T02:21:53.764 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247810 2026-02-13T02:21:53.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247810' 2026-02-13T02:21:53.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:21:53.765 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:21:53.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=85899345922 2026-02-13T02:21:53.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 85899345922 2026-02-13T02:21:53.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247810 1-85899345922' 2026-02-13T02:21:53.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:21:53.843 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-98784247810 2026-02-13T02:21:53.843 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:21:53.845 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:21:53.846 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-98784247810 2026-02-13T02:21:53.846 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:21:53.847 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247810 2026-02-13T02:21:53.848 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 98784247810 2026-02-13T02:21:53.848 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 98784247810' 2026-02-13T02:21:53.848 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:21:54.017 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247810 -lt 98784247810 2026-02-13T02:21:54.017 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:21:54.018 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-85899345922 2026-02-13T02:21:54.018 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:21:54.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:21:54.020 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-85899345922 2026-02-13T02:21:54.021 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:21:54.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=85899345922 2026-02-13T02:21:54.023 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 85899345922 2026-02-13T02:21:54.023 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 85899345922' 2026-02-13T02:21:54.023 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:21:54.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 85899345922 -lt 85899345922 2026-02-13T02:21:54.192 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:21:54.192 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:21:54.192 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:21:54.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:21:54.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:21:54.389 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:21:54.389 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:21:54.389 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:21:54.389 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:21:54.389 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:21:54.389 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:21:54.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:21:54.547 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:21:54.547 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:21:54.547 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:21:54.747 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:21:54.747 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:21:54.747 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:21:54.747 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:759: TEST_repair_stats: repair 1.0 2026-02-13T02:21:54.747 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=1.0 2026-02-13T02:21:54.747 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 1.0 2026-02-13T02:21:54.747 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:21:54.748 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:21:54.748 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:21:54.748 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:21:54.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-13T02:21:39.058489+0000 2026-02-13T02:21:54.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 1.0 2026-02-13T02:21:55.057 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 1.0 on osd.1 to repair 2026-02-13T02:21:55.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 1.0 2026-02-13T02:21:39.058489+0000 2026-02-13T02:21:55.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-13T02:21:55.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:21:39.058489+0000 2026-02-13T02:21:55.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:21:55.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:21:55.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:21:55.068 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:21:55.068 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:21:55.068 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:21:55.069 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:21:55.069 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:21:55.229 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:21:39.058489+0000 '>' 2026-02-13T02:21:39.058489+0000 2026-02-13T02:21:55.230 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:21:56.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:21:56.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:21:56.232 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:21:56.232 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:21:56.232 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:21:56.232 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:21:56.233 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:21:56.395 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:21:39.058489+0000 '>' 2026-02-13T02:21:39.058489+0000 2026-02-13T02:21:56.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:21:57.397 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:21:57.397 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:21:57.398 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:21:57.398 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:21:57.398 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:21:57.399 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:21:57.399 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:21:57.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:21:39.058489+0000 '>' 2026-02-13T02:21:39.058489+0000 2026-02-13T02:21:57.556 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:21:58.556 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:21:58.556 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:21:58.557 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:21:58.557 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:21:58.557 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:21:58.557 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:21:58.558 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:21:58.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:21:55.511227+0000 '>' 2026-02-13T02:21:39.058489+0000 2026-02-13T02:21:58.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:21:58.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats: wait_for_clean 2026-02-13T02:21:58.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:21:58.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:21:58.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:21:58.715 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:21:58.716 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:21:58.716 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:21:58.716 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:21:58.716 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:21:58.716 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:21:58.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:21:58.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:21:58.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:21:58.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:21:58.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:21:58.830 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:21:58.998 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:21:58.998 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:21:58.998 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:21:58.998 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:21:58.998 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:21:59.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247812 2026-02-13T02:21:59.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247812 2026-02-13T02:21:59.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247812' 2026-02-13T02:21:59.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:21:59.071 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:21:59.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=85899345924 2026-02-13T02:21:59.144 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 85899345924 2026-02-13T02:21:59.144 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247812 1-85899345924' 2026-02-13T02:21:59.144 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:21:59.144 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-98784247812 2026-02-13T02:21:59.145 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:21:59.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:21:59.147 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-98784247812 2026-02-13T02:21:59.147 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:21:59.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247812 2026-02-13T02:21:59.149 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 98784247812 2026-02-13T02:21:59.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 98784247812' 2026-02-13T02:21:59.149 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:21:59.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247810 -lt 98784247812 2026-02-13T02:21:59.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:22:00.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:22:00.306 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:22:00.463 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247812 -lt 98784247812 2026-02-13T02:22:00.463 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:22:00.464 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-85899345924 2026-02-13T02:22:00.464 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:22:00.465 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:22:00.466 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-85899345924 2026-02-13T02:22:00.466 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:22:00.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=85899345924 2026-02-13T02:22:00.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 85899345924' 2026-02-13T02:22:00.468 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 85899345924 2026-02-13T02:22:00.468 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:22:00.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 85899345924 -lt 85899345924 2026-02-13T02:22:00.630 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:22:00.631 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:22:00.631 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:22:00.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:22:00.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:22:00.827 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:22:00.827 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:22:00.827 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:22:00.828 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:22:00.828 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:22:00.828 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:22:00.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:22:00.984 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:22:00.984 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:22:00.985 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:22:01.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:22:01.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:22:01.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:22:01.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats: ceph pg dump pgs 2026-02-13T02:22:01.324 INFO:tasks.workunit.client.0.trial112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-13T02:22:01.324 INFO:tasks.workunit.client.0.trial112.stdout:1.0 30 0 0 0 0 210 0 0 30 0 30 active+clean 2026-02-13T02:21:55.527832+0000 15'30 25:132 [1,0] 1 [1,0] 1 15'30 2026-02-13T02:21:55.511227+0000 15'30 2026-02-13T02:21:55.511227+0000 0 1 periodic scrub scheduled @ 2026-02-14T02:21:55.511227+0000 20 0 2026-02-13T02:22:01.324 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T02:22:01.324 INFO:tasks.workunit.client.0.trial112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-13T02:22:01.325 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:22:01.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:762: TEST_repair_stats: flush_pg_stats 2026-02-13T02:22:01.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:22:01.334 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:22:01.513 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:22:01.513 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:22:01.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:22:01.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:22:01.514 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:22:01.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247813 2026-02-13T02:22:01.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247813 2026-02-13T02:22:01.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247813' 2026-02-13T02:22:01.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:22:01.588 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:22:01.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=85899345926 2026-02-13T02:22:01.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 85899345926 2026-02-13T02:22:01.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247813 1-85899345926' 2026-02-13T02:22:01.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:22:01.662 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-98784247813 2026-02-13T02:22:01.662 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:22:01.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:22:01.664 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-98784247813 2026-02-13T02:22:01.664 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:22:01.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247813 2026-02-13T02:22:01.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 98784247813' 2026-02-13T02:22:01.666 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 98784247813 2026-02-13T02:22:01.667 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:22:01.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247812 -lt 98784247813 2026-02-13T02:22:01.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:22:02.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:22:02.842 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:22:03.001 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247813 -lt 98784247813 2026-02-13T02:22:03.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:22:03.002 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-85899345926 2026-02-13T02:22:03.002 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:22:03.004 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:22:03.005 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-85899345926 2026-02-13T02:22:03.005 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:22:03.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=85899345926 2026-02-13T02:22:03.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 85899345926' 2026-02-13T02:22:03.007 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 85899345926 2026-02-13T02:22:03.007 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:22:03.184 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 85899345926 -lt 85899345926 2026-02-13T02:22:03.184 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:765: TEST_repair_stats: ceph pg 1.0 query 2026-02-13T02:22:03.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:765: TEST_repair_stats: jq .info.stats.stat_sum 2026-02-13T02:22:03.257 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:22:03.257 INFO:tasks.workunit.client.0.trial112.stdout: "num_bytes": 210, 2026-02-13T02:22:03.257 INFO:tasks.workunit.client.0.trial112.stdout: "num_objects": 30, 2026-02-13T02:22:03.257 INFO:tasks.workunit.client.0.trial112.stdout: "num_object_clones": 0, 2026-02-13T02:22:03.258 INFO:tasks.workunit.client.0.trial112.stdout: "num_object_copies": 60, 2026-02-13T02:22:03.258 INFO:tasks.workunit.client.0.trial112.stdout: "num_objects_missing_on_primary": 0, 2026-02-13T02:22:03.258 INFO:tasks.workunit.client.0.trial112.stdout: "num_objects_missing": 0, 2026-02-13T02:22:03.258 INFO:tasks.workunit.client.0.trial112.stdout: "num_objects_degraded": 0, 2026-02-13T02:22:03.258 INFO:tasks.workunit.client.0.trial112.stdout: "num_objects_misplaced": 0, 2026-02-13T02:22:03.258 INFO:tasks.workunit.client.0.trial112.stdout: "num_objects_unfound": 0, 2026-02-13T02:22:03.258 INFO:tasks.workunit.client.0.trial112.stdout: "num_objects_dirty": 30, 2026-02-13T02:22:03.258 INFO:tasks.workunit.client.0.trial112.stdout: "num_whiteouts": 0, 2026-02-13T02:22:03.258 INFO:tasks.workunit.client.0.trial112.stdout: "num_read": 0, 2026-02-13T02:22:03.258 INFO:tasks.workunit.client.0.trial112.stdout: "num_read_kb": 0, 2026-02-13T02:22:03.258 INFO:tasks.workunit.client.0.trial112.stdout: "num_write": 30, 2026-02-13T02:22:03.258 INFO:tasks.workunit.client.0.trial112.stdout: "num_write_kb": 30, 2026-02-13T02:22:03.259 INFO:tasks.workunit.client.0.trial112.stdout: "num_scrub_errors": 0, 2026-02-13T02:22:03.259 INFO:tasks.workunit.client.0.trial112.stdout: "num_shallow_scrub_errors": 0, 2026-02-13T02:22:03.259 INFO:tasks.workunit.client.0.trial112.stdout: "num_deep_scrub_errors": 0, 2026-02-13T02:22:03.259 INFO:tasks.workunit.client.0.trial112.stdout: "num_objects_recovered": 20, 2026-02-13T02:22:03.259 INFO:tasks.workunit.client.0.trial112.stdout: "num_bytes_recovered": 140, 2026-02-13T02:22:03.259 INFO:tasks.workunit.client.0.trial112.stdout: "num_keys_recovered": 0, 2026-02-13T02:22:03.259 INFO:tasks.workunit.client.0.trial112.stdout: "num_objects_omap": 0, 2026-02-13T02:22:03.259 INFO:tasks.workunit.client.0.trial112.stdout: "num_objects_hit_set_archive": 0, 2026-02-13T02:22:03.259 INFO:tasks.workunit.client.0.trial112.stdout: "num_bytes_hit_set_archive": 0, 2026-02-13T02:22:03.259 INFO:tasks.workunit.client.0.trial112.stdout: "num_flush": 0, 2026-02-13T02:22:03.259 INFO:tasks.workunit.client.0.trial112.stdout: "num_flush_kb": 0, 2026-02-13T02:22:03.259 INFO:tasks.workunit.client.0.trial112.stdout: "num_evict": 0, 2026-02-13T02:22:03.259 INFO:tasks.workunit.client.0.trial112.stdout: "num_evict_kb": 0, 2026-02-13T02:22:03.260 INFO:tasks.workunit.client.0.trial112.stdout: "num_promote": 0, 2026-02-13T02:22:03.260 INFO:tasks.workunit.client.0.trial112.stdout: "num_flush_mode_high": 0, 2026-02-13T02:22:03.260 INFO:tasks.workunit.client.0.trial112.stdout: "num_flush_mode_low": 0, 2026-02-13T02:22:03.260 INFO:tasks.workunit.client.0.trial112.stdout: "num_evict_mode_some": 0, 2026-02-13T02:22:03.260 INFO:tasks.workunit.client.0.trial112.stdout: "num_evict_mode_full": 0, 2026-02-13T02:22:03.260 INFO:tasks.workunit.client.0.trial112.stdout: "num_objects_pinned": 0, 2026-02-13T02:22:03.260 INFO:tasks.workunit.client.0.trial112.stdout: "num_legacy_snapsets": 0, 2026-02-13T02:22:03.260 INFO:tasks.workunit.client.0.trial112.stdout: "num_large_omap_objects": 0, 2026-02-13T02:22:03.260 INFO:tasks.workunit.client.0.trial112.stdout: "num_objects_manifest": 0, 2026-02-13T02:22:03.260 INFO:tasks.workunit.client.0.trial112.stdout: "num_omap_bytes": 0, 2026-02-13T02:22:03.260 INFO:tasks.workunit.client.0.trial112.stdout: "num_omap_keys": 0, 2026-02-13T02:22:03.260 INFO:tasks.workunit.client.0.trial112.stdout: "num_objects_repaired": 20 2026-02-13T02:22:03.261 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:22:03.261 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:766: TEST_repair_stats: ceph pg 1.0 query 2026-02-13T02:22:03.261 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:766: TEST_repair_stats: jq .info.stats.stat_sum.num_objects_repaired 2026-02-13T02:22:03.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:766: TEST_repair_stats: COUNT=20 2026-02-13T02:22:03.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:767: TEST_repair_stats: test 20 = 20 2026-02-13T02:22:03.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:769: TEST_repair_stats: ceph pg dump --format=json-pretty 2026-02-13T02:22:03.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:769: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 1 )' 2026-02-13T02:22:03.482 INFO:tasks.workunit.client.0.trial112.stderr:dumped all 2026-02-13T02:22:03.491 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:22:03.491 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 1, 2026-02-13T02:22:03.491 INFO:tasks.workunit.client.0.trial112.stdout: "up_from": 20, 2026-02-13T02:22:03.491 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 85899345926, 2026-02-13T02:22:03.491 INFO:tasks.workunit.client.0.trial112.stdout: "num_pgs": 1, 2026-02-13T02:22:03.491 INFO:tasks.workunit.client.0.trial112.stdout: "num_osds": 1, 2026-02-13T02:22:03.491 INFO:tasks.workunit.client.0.trial112.stdout: "num_per_pool_osds": 1, 2026-02-13T02:22:03.491 INFO:tasks.workunit.client.0.trial112.stdout: "num_per_pool_omap_osds": 1, 2026-02-13T02:22:03.491 INFO:tasks.workunit.client.0.trial112.stdout: "kb": 104857600, 2026-02-13T02:22:03.491 INFO:tasks.workunit.client.0.trial112.stdout: "kb_used": 28568, 2026-02-13T02:22:03.491 INFO:tasks.workunit.client.0.trial112.stdout: "kb_used_data": 328, 2026-02-13T02:22:03.492 INFO:tasks.workunit.client.0.trial112.stdout: "kb_used_omap": 8, 2026-02-13T02:22:03.492 INFO:tasks.workunit.client.0.trial112.stdout: "kb_used_meta": 28215, 2026-02-13T02:22:03.492 INFO:tasks.workunit.client.0.trial112.stdout: "kb_avail": 104829032, 2026-02-13T02:22:03.492 INFO:tasks.workunit.client.0.trial112.stdout: "statfs": { 2026-02-13T02:22:03.492 INFO:tasks.workunit.client.0.trial112.stdout: "total": 107374182400, 2026-02-13T02:22:03.492 INFO:tasks.workunit.client.0.trial112.stdout: "available": 107344928768, 2026-02-13T02:22:03.492 INFO:tasks.workunit.client.0.trial112.stdout: "internally_reserved": 0, 2026-02-13T02:22:03.492 INFO:tasks.workunit.client.0.trial112.stdout: "allocated": 335872, 2026-02-13T02:22:03.492 INFO:tasks.workunit.client.0.trial112.stdout: "data_stored": 56545, 2026-02-13T02:22:03.492 INFO:tasks.workunit.client.0.trial112.stdout: "data_compressed": 0, 2026-02-13T02:22:03.492 INFO:tasks.workunit.client.0.trial112.stdout: "data_compressed_allocated": 0, 2026-02-13T02:22:03.492 INFO:tasks.workunit.client.0.trial112.stdout: "data_compressed_original": 0, 2026-02-13T02:22:03.493 INFO:tasks.workunit.client.0.trial112.stdout: "omap_allocated": 8310, 2026-02-13T02:22:03.493 INFO:tasks.workunit.client.0.trial112.stdout: "internal_metadata": 28893066 2026-02-13T02:22:03.493 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:22:03.493 INFO:tasks.workunit.client.0.trial112.stdout: "hb_peers": [ 2026-02-13T02:22:03.493 INFO:tasks.workunit.client.0.trial112.stdout: 0 2026-02-13T02:22:03.493 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:22:03.493 INFO:tasks.workunit.client.0.trial112.stdout: "snap_trim_queue_len": 0, 2026-02-13T02:22:03.493 INFO:tasks.workunit.client.0.trial112.stdout: "num_snap_trimming": 0, 2026-02-13T02:22:03.493 INFO:tasks.workunit.client.0.trial112.stdout: "num_shards_repaired": 10, 2026-02-13T02:22:03.493 INFO:tasks.workunit.client.0.trial112.stdout: "op_queue_age_hist": { 2026-02-13T02:22:03.493 INFO:tasks.workunit.client.0.trial112.stdout: "histogram": [], 2026-02-13T02:22:03.493 INFO:tasks.workunit.client.0.trial112.stdout: "upper_bound": 1 2026-02-13T02:22:03.494 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:22:03.494 INFO:tasks.workunit.client.0.trial112.stdout: "perf_stat": { 2026-02-13T02:22:03.494 INFO:tasks.workunit.client.0.trial112.stdout: "commit_latency_ms": 0, 2026-02-13T02:22:03.494 INFO:tasks.workunit.client.0.trial112.stdout: "apply_latency_ms": 0, 2026-02-13T02:22:03.494 INFO:tasks.workunit.client.0.trial112.stdout: "commit_latency_ns": 0, 2026-02-13T02:22:03.494 INFO:tasks.workunit.client.0.trial112.stdout: "apply_latency_ns": 0 2026-02-13T02:22:03.494 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:22:03.494 INFO:tasks.workunit.client.0.trial112.stdout: "alerts": [] 2026-02-13T02:22:03.494 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:22:03.495 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:770: TEST_repair_stats: ceph pg dump --format=json-pretty 2026-02-13T02:22:03.495 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:770: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 1 ).num_shards_repaired' 2026-02-13T02:22:03.645 INFO:tasks.workunit.client.0.trial112.stderr:dumped all 2026-02-13T02:22:03.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:770: TEST_repair_stats: COUNT=10 2026-02-13T02:22:03.655 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:771: TEST_repair_stats: expr 20 / 2 2026-02-13T02:22:03.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:771: TEST_repair_stats: test 10 = 10 2026-02-13T02:22:03.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:773: TEST_repair_stats: ceph pg dump --format=json-pretty 2026-02-13T02:22:03.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:773: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 0 )' 2026-02-13T02:22:03.805 INFO:tasks.workunit.client.0.trial112.stderr:dumped all 2026-02-13T02:22:03.816 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:22:03.816 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 0, 2026-02-13T02:22:03.816 INFO:tasks.workunit.client.0.trial112.stdout: "up_from": 23, 2026-02-13T02:22:03.816 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 98784247813, 2026-02-13T02:22:03.816 INFO:tasks.workunit.client.0.trial112.stdout: "num_pgs": 1, 2026-02-13T02:22:03.816 INFO:tasks.workunit.client.0.trial112.stdout: "num_osds": 1, 2026-02-13T02:22:03.816 INFO:tasks.workunit.client.0.trial112.stdout: "num_per_pool_osds": 1, 2026-02-13T02:22:03.816 INFO:tasks.workunit.client.0.trial112.stdout: "num_per_pool_omap_osds": 1, 2026-02-13T02:22:03.816 INFO:tasks.workunit.client.0.trial112.stdout: "kb": 104857600, 2026-02-13T02:22:03.817 INFO:tasks.workunit.client.0.trial112.stdout: "kb_used": 28568, 2026-02-13T02:22:03.817 INFO:tasks.workunit.client.0.trial112.stdout: "kb_used_data": 328, 2026-02-13T02:22:03.817 INFO:tasks.workunit.client.0.trial112.stdout: "kb_used_omap": 6, 2026-02-13T02:22:03.817 INFO:tasks.workunit.client.0.trial112.stdout: "kb_used_meta": 28217, 2026-02-13T02:22:03.817 INFO:tasks.workunit.client.0.trial112.stdout: "kb_avail": 104829032, 2026-02-13T02:22:03.817 INFO:tasks.workunit.client.0.trial112.stdout: "statfs": { 2026-02-13T02:22:03.817 INFO:tasks.workunit.client.0.trial112.stdout: "total": 107374182400, 2026-02-13T02:22:03.817 INFO:tasks.workunit.client.0.trial112.stdout: "available": 107344928768, 2026-02-13T02:22:03.817 INFO:tasks.workunit.client.0.trial112.stdout: "internally_reserved": 0, 2026-02-13T02:22:03.817 INFO:tasks.workunit.client.0.trial112.stdout: "allocated": 335872, 2026-02-13T02:22:03.817 INFO:tasks.workunit.client.0.trial112.stdout: "data_stored": 56545, 2026-02-13T02:22:03.818 INFO:tasks.workunit.client.0.trial112.stdout: "data_compressed": 0, 2026-02-13T02:22:03.818 INFO:tasks.workunit.client.0.trial112.stdout: "data_compressed_allocated": 0, 2026-02-13T02:22:03.818 INFO:tasks.workunit.client.0.trial112.stdout: "data_compressed_original": 0, 2026-02-13T02:22:03.818 INFO:tasks.workunit.client.0.trial112.stdout: "omap_allocated": 6354, 2026-02-13T02:22:03.818 INFO:tasks.workunit.client.0.trial112.stdout: "internal_metadata": 28895022 2026-02-13T02:22:03.818 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:22:03.818 INFO:tasks.workunit.client.0.trial112.stdout: "hb_peers": [ 2026-02-13T02:22:03.818 INFO:tasks.workunit.client.0.trial112.stdout: 1 2026-02-13T02:22:03.818 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:22:03.818 INFO:tasks.workunit.client.0.trial112.stdout: "snap_trim_queue_len": 0, 2026-02-13T02:22:03.818 INFO:tasks.workunit.client.0.trial112.stdout: "num_snap_trimming": 0, 2026-02-13T02:22:03.818 INFO:tasks.workunit.client.0.trial112.stdout: "num_shards_repaired": 10, 2026-02-13T02:22:03.819 INFO:tasks.workunit.client.0.trial112.stdout: "op_queue_age_hist": { 2026-02-13T02:22:03.819 INFO:tasks.workunit.client.0.trial112.stdout: "histogram": [], 2026-02-13T02:22:03.819 INFO:tasks.workunit.client.0.trial112.stdout: "upper_bound": 1 2026-02-13T02:22:03.819 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:22:03.819 INFO:tasks.workunit.client.0.trial112.stdout: "perf_stat": { 2026-02-13T02:22:03.819 INFO:tasks.workunit.client.0.trial112.stdout: "commit_latency_ms": 0, 2026-02-13T02:22:03.819 INFO:tasks.workunit.client.0.trial112.stdout: "apply_latency_ms": 0, 2026-02-13T02:22:03.819 INFO:tasks.workunit.client.0.trial112.stdout: "commit_latency_ns": 0, 2026-02-13T02:22:03.819 INFO:tasks.workunit.client.0.trial112.stdout: "apply_latency_ns": 0 2026-02-13T02:22:03.819 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:22:03.819 INFO:tasks.workunit.client.0.trial112.stdout: "alerts": [] 2026-02-13T02:22:03.819 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:22:03.820 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:774: TEST_repair_stats: ceph pg dump --format=json-pretty 2026-02-13T02:22:03.820 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:774: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 0 ).num_shards_repaired' 2026-02-13T02:22:03.962 INFO:tasks.workunit.client.0.trial112.stderr:dumped all 2026-02-13T02:22:03.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:774: TEST_repair_stats: COUNT=10 2026-02-13T02:22:03.973 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:775: TEST_repair_stats: expr 20 / 2 2026-02-13T02:22:03.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:775: TEST_repair_stats: test 10 = 10 2026-02-13T02:22:03.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats: ceph pg dump --format=json-pretty 2026-02-13T02:22:03.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats: jq .pg_map.osd_stats_sum 2026-02-13T02:22:04.121 INFO:tasks.workunit.client.0.trial112.stderr:dumped all 2026-02-13T02:22:04.130 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:22:04.130 INFO:tasks.workunit.client.0.trial112.stdout: "up_from": 0, 2026-02-13T02:22:04.130 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 0, 2026-02-13T02:22:04.130 INFO:tasks.workunit.client.0.trial112.stdout: "num_pgs": 2, 2026-02-13T02:22:04.130 INFO:tasks.workunit.client.0.trial112.stdout: "num_osds": 2, 2026-02-13T02:22:04.131 INFO:tasks.workunit.client.0.trial112.stdout: "num_per_pool_osds": 2, 2026-02-13T02:22:04.131 INFO:tasks.workunit.client.0.trial112.stdout: "num_per_pool_omap_osds": 2, 2026-02-13T02:22:04.131 INFO:tasks.workunit.client.0.trial112.stdout: "kb": 209715200, 2026-02-13T02:22:04.131 INFO:tasks.workunit.client.0.trial112.stdout: "kb_used": 57136, 2026-02-13T02:22:04.131 INFO:tasks.workunit.client.0.trial112.stdout: "kb_used_data": 656, 2026-02-13T02:22:04.131 INFO:tasks.workunit.client.0.trial112.stdout: "kb_used_omap": 14, 2026-02-13T02:22:04.131 INFO:tasks.workunit.client.0.trial112.stdout: "kb_used_meta": 56433, 2026-02-13T02:22:04.131 INFO:tasks.workunit.client.0.trial112.stdout: "kb_avail": 209658064, 2026-02-13T02:22:04.131 INFO:tasks.workunit.client.0.trial112.stdout: "statfs": { 2026-02-13T02:22:04.131 INFO:tasks.workunit.client.0.trial112.stdout: "total": 214748364800, 2026-02-13T02:22:04.132 INFO:tasks.workunit.client.0.trial112.stdout: "available": 214689857536, 2026-02-13T02:22:04.132 INFO:tasks.workunit.client.0.trial112.stdout: "internally_reserved": 0, 2026-02-13T02:22:04.132 INFO:tasks.workunit.client.0.trial112.stdout: "allocated": 671744, 2026-02-13T02:22:04.132 INFO:tasks.workunit.client.0.trial112.stdout: "data_stored": 113090, 2026-02-13T02:22:04.132 INFO:tasks.workunit.client.0.trial112.stdout: "data_compressed": 0, 2026-02-13T02:22:04.132 INFO:tasks.workunit.client.0.trial112.stdout: "data_compressed_allocated": 0, 2026-02-13T02:22:04.132 INFO:tasks.workunit.client.0.trial112.stdout: "data_compressed_original": 0, 2026-02-13T02:22:04.132 INFO:tasks.workunit.client.0.trial112.stdout: "omap_allocated": 14664, 2026-02-13T02:22:04.132 INFO:tasks.workunit.client.0.trial112.stdout: "internal_metadata": 57788088 2026-02-13T02:22:04.132 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:22:04.132 INFO:tasks.workunit.client.0.trial112.stdout: "hb_peers": [], 2026-02-13T02:22:04.132 INFO:tasks.workunit.client.0.trial112.stdout: "snap_trim_queue_len": 0, 2026-02-13T02:22:04.133 INFO:tasks.workunit.client.0.trial112.stdout: "num_snap_trimming": 0, 2026-02-13T02:22:04.133 INFO:tasks.workunit.client.0.trial112.stdout: "num_shards_repaired": 20, 2026-02-13T02:22:04.133 INFO:tasks.workunit.client.0.trial112.stdout: "op_queue_age_hist": { 2026-02-13T02:22:04.133 INFO:tasks.workunit.client.0.trial112.stdout: "histogram": [], 2026-02-13T02:22:04.133 INFO:tasks.workunit.client.0.trial112.stdout: "upper_bound": 1 2026-02-13T02:22:04.133 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:22:04.133 INFO:tasks.workunit.client.0.trial112.stdout: "perf_stat": { 2026-02-13T02:22:04.133 INFO:tasks.workunit.client.0.trial112.stdout: "commit_latency_ms": 0, 2026-02-13T02:22:04.133 INFO:tasks.workunit.client.0.trial112.stdout: "apply_latency_ms": 0, 2026-02-13T02:22:04.133 INFO:tasks.workunit.client.0.trial112.stdout: "commit_latency_ns": 0, 2026-02-13T02:22:04.133 INFO:tasks.workunit.client.0.trial112.stdout: "apply_latency_ns": 0 2026-02-13T02:22:04.133 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:22:04.134 INFO:tasks.workunit.client.0.trial112.stdout: "alerts": [], 2026-02-13T02:22:04.134 INFO:tasks.workunit.client.0.trial112.stdout: "network_ping_times": [] 2026-02-13T02:22:04.134 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:22:04.134 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:778: TEST_repair_stats: ceph pg dump --format=json-pretty 2026-02-13T02:22:04.135 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:778: TEST_repair_stats: jq .pg_map.osd_stats_sum.num_shards_repaired 2026-02-13T02:22:04.284 INFO:tasks.workunit.client.0.trial112.stderr:dumped all 2026-02-13T02:22:04.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:778: TEST_repair_stats: COUNT=20 2026-02-13T02:22:04.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:779: TEST_repair_stats: test 20 = 20 2026-02-13T02:22:04.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-13T02:22:04.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:22:04.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:22:04.295 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:22:04.296 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:22:04.296 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:22:04.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:22:04.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:22:04.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:22:04.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:22:04.607 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:22:04.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:22:04.609 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:22:04.611 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:22:04.611 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:22:04.612 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:22:04.613 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:22:04.613 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:22:04.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:22:04.614 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:22:04.615 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:22:04.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:22:04.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:22:04.628 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:22:04.629 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:22:04.629 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:22:04.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:22:04.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:22:04.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:22:04.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-13T02:22:04.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-13T02:22:04.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-13T02:22:04.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-13T02:22:04.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:22:04.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:22:04.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:22:04.632 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:22:04.632 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:22:04.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:22:04.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:22:04.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:22:04.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:22:04.635 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:22:04.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:22:04.637 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:22:04.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:22:04.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:22:04.639 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:22:04.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:22:04.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:22:04.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:22:04.641 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:22:04.642 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:22:04.644 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:22:04.644 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:22:04.645 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:22:04.645 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:22:04.645 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:22:04.646 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:22:04.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:22:04.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:22:04.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-13T02:22:04.648 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T02:22:04.649 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:22:04.649 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:22:04.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.35207 2026-02-13T02:22:04.651 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T02:22:04.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-13T02:22:04.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T02:22:04.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-13T02:22:04.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_repair_stats_ec td/osd-scrub-repair 2026-02-13T02:22:04.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:783: TEST_repair_stats_ec: local dir=td/osd-scrub-repair 2026-02-13T02:22:04.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:784: TEST_repair_stats_ec: local poolname=testpool 2026-02-13T02:22:04.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:785: TEST_repair_stats_ec: local OSDS=3 2026-02-13T02:22:04.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:786: TEST_repair_stats_ec: local OBJS=30 2026-02-13T02:22:04.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: local REPAIRS=26 2026-02-13T02:22:04.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: local allow_overwrites=false 2026-02-13T02:22:04.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:792: TEST_repair_stats_ec: run_mon td/osd-scrub-repair a 2026-02-13T02:22:04.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-13T02:22:04.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-13T02:22:04.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-13T02:22:04.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-13T02:22:04.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-13T02:22:04.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-13T02:22:04.674 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-13T02:22:04.674 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:22:04.674 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:22:04.674 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:22:04.674 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:22:04.674 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:22:04.675 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:22:04.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-13T02:22:04.705 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-13T02:22:04.706 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-13T02:22:04.707 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:22:04.707 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:22:04.707 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-13T02:22:04.707 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-13T02:22:04.708 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:22:04.708 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:22:04.708 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:22:04.709 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:22:04.709 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:22:04.709 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:22:04.709 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:22:04.709 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:22:04.709 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get fsid 2026-02-13T02:22:04.765 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-13T02:22:04.765 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:22:04.765 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:22:04.765 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-13T02:22:04.765 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-13T02:22:04.766 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:22:04.766 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:22:04.766 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:22:04.766 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:22:04.766 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:22:04.766 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:22:04.766 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:22:04.767 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:22:04.767 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get mon_host 2026-02-13T02:22:04.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:793: TEST_repair_stats_ec: run_mgr td/osd-scrub-repair x 2026-02-13T02:22:04.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-13T02:22:04.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-13T02:22:04.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-13T02:22:04.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-13T02:22:04.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-13T02:22:04.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-13T02:22:04.920 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-13T02:22:04.920 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:22:04.920 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:22:04.920 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:22:04.920 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:22:04.920 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:22:04.920 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:22:04.921 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:22:04.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:22:04.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:795: TEST_repair_stats_ec: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-13T02:22:04.940 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:796: TEST_repair_stats_ec: expr 3 - 1 2026-02-13T02:22:04.941 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:796: TEST_repair_stats_ec: seq 0 2 2026-02-13T02:22:04.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:796: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:22:04.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:797: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T02:22:04.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:22:04.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:22:04.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-13T02:22:04.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:22:04.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:22:04.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:22:04.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:22:04.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:22:04.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:22:04.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:22:04.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:22:04.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:22:04.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:22:04.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:22:04.944 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:22:04.944 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:22:04.945 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:22:04.945 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:22:04.945 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:22:04.945 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:22:04.945 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:22:04.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:22:04.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:22:04.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:22:04.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:22:04.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:22:04.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:22:04.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:22:04.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:22:04.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:22:04.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:22:04.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:22:04.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-13T02:22:04.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:22:04.946 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:22:04.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a2bceff7-8c06-46ae-bc7a-5b0e7f1db95e 2026-02-13T02:22:04.947 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 a2bceff7-8c06-46ae-bc7a-5b0e7f1db95e 2026-02-13T02:22:04.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 a2bceff7-8c06-46ae-bc7a-5b0e7f1db95e' 2026-02-13T02:22:04.948 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:22:04.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDMio5p0fODOBAA6QEnBkpM6OtyqviokgPgVg== 2026-02-13T02:22:04.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDMio5p0fODOBAA6QEnBkpM6OtyqviokgPgVg=="}' 2026-02-13T02:22:04.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a2bceff7-8c06-46ae-bc7a-5b0e7f1db95e -i td/osd-scrub-repair/0/new.json 2026-02-13T02:22:05.049 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:22:05.058 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-13T02:22:05.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDMio5p0fODOBAA6QEnBkpM6OtyqviokgPgVg== --osd-uuid a2bceff7-8c06-46ae-bc7a-5b0e7f1db95e 2026-02-13T02:22:05.076 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:05.059+0000 7f1f60c67a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:22:05.077 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:05.063+0000 7f1f60c67a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:22:05.078 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:05.063+0000 7f1f60c67a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:22:05.078 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:05.063+0000 7f1f60c67a40 -1 bdev(0x5653fb602c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-13T02:22:05.079 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:05.063+0000 7f1f60c67a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-13T02:22:05.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-13T02:22:05.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:22:05.757 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T02:22:05.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-13T02:22:05.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:22:05.857 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T02:22:05.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-13T02:22:05.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T02:22:05.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:22:05.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:22:05.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:22:05.872 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:05.859+0000 7fd93289da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:22:05.874 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:05.859+0000 7fd93289da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:22:05.875 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:05.859+0000 7fd93289da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:22:05.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-13T02:22:05.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:22:05.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:22:05.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:22:05.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:22:05.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:22:05.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:22:05.952 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:22:05.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:22:05.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:22:06.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:22:06.258 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:06.243+0000 7fd93289da40 -1 Falling back to public interface 2026-02-13T02:22:06.890 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:06.875+0000 7fd93289da40 -1 osd.0 0 log_to_monitors true 2026-02-13T02:22:07.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:22:07.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:22:07.046 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:22:07.047 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:22:07.048 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:22:07.052 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:22:07.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:22:08.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:22:08.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:22:08.186 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:22:08.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:22:08.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:22:08.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:22:08.345 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3583713821,v1:127.0.0.1:6801/3583713821] [v2:127.0.0.1:6802/3583713821,v1:127.0.0.1:6803/3583713821] exists,up a2bceff7-8c06-46ae-bc7a-5b0e7f1db95e 2026-02-13T02:22:08.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:22:08.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:22:08.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:22:08.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:796: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:22:08.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:797: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T02:22:08.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:22:08.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:22:08.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-13T02:22:08.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:22:08.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:22:08.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:22:08.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:22:08.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:22:08.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:22:08.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:22:08.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:22:08.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:22:08.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:22:08.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:22:08.350 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:22:08.350 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:22:08.351 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:22:08.351 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:22:08.351 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:22:08.351 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:22:08.351 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:22:08.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:22:08.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:22:08.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:22:08.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:22:08.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:22:08.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:22:08.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:22:08.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:22:08.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:22:08.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:22:08.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:22:08.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-13T02:22:08.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:22:08.353 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:22:08.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5a27e272-65c7-49e4-a7e7-27bfe069e85a 2026-02-13T02:22:08.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 5a27e272-65c7-49e4-a7e7-27bfe069e85a' 2026-02-13T02:22:08.353 INFO:tasks.workunit.client.0.trial112.stdout:add osd1 5a27e272-65c7-49e4-a7e7-27bfe069e85a 2026-02-13T02:22:08.353 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:22:08.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDQio5pXlkTFRAArWCzLT+CRMa6/5ARPMNiSQ== 2026-02-13T02:22:08.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDQio5pXlkTFRAArWCzLT+CRMa6/5ARPMNiSQ=="}' 2026-02-13T02:22:08.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5a27e272-65c7-49e4-a7e7-27bfe069e85a -i td/osd-scrub-repair/1/new.json 2026-02-13T02:22:08.527 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:22:08.536 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-13T02:22:08.537 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDQio5pXlkTFRAArWCzLT+CRMa6/5ARPMNiSQ== --osd-uuid 5a27e272-65c7-49e4-a7e7-27bfe069e85a 2026-02-13T02:22:08.554 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:08.539+0000 7fe7915f8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:22:08.555 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:08.539+0000 7fe7915f8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:22:08.556 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:08.543+0000 7fe7915f8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:22:08.557 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:08.543+0000 7fe7915f8a40 -1 bdev(0x561cd5f10c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-13T02:22:08.557 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:08.543+0000 7fe7915f8a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-13T02:22:09.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-13T02:22:09.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:22:09.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-13T02:22:09.173 INFO:tasks.workunit.client.0.trial112.stdout:adding osd1 key to auth repository 2026-02-13T02:22:09.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:22:09.381 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-13T02:22:09.381 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T02:22:09.382 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T02:22:09.382 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:22:09.382 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:22:09.382 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:22:09.405 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:09.391+0000 7f1999a9da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:22:09.406 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:09.391+0000 7f1999a9da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:22:09.407 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:09.391+0000 7f1999a9da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:22:09.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-13T02:22:09.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:22:09.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:22:09.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:22:09.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:22:09.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:22:09.546 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:22:09.547 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:22:09.547 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:22:09.547 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:22:09.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:22:09.802 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:09.787+0000 7f1999a9da40 -1 Falling back to public interface 2026-02-13T02:22:10.452 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:10.435+0000 7f1999a9da40 -1 osd.1 0 log_to_monitors true 2026-02-13T02:22:10.707 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:22:10.707 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:22:10.707 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:22:10.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:22:10.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:22:10.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:22:10.870 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:22:11.871 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:22:11.871 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:22:11.872 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:22:11.872 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:22:11.872 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:22:11.872 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:22:12.036 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2639897229,v1:127.0.0.1:6811/2639897229] [v2:127.0.0.1:6812/2639897229,v1:127.0.0.1:6813/2639897229] exists,up 5a27e272-65c7-49e4-a7e7-27bfe069e85a 2026-02-13T02:22:12.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:22:12.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:22:12.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:22:12.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:796: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:22:12.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:797: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 2 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T02:22:12.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:22:12.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:22:12.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-13T02:22:12.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:22:12.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-13T02:22:12.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:22:12.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:22:12.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:22:12.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:22:12.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-13T02:22:12.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-13T02:22:12.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:22:12.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:22:12.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:22:12.040 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:22:12.040 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:22:12.041 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:22:12.041 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:22:12.041 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:22:12.041 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:22:12.041 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:22:12.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:22:12.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:22:12.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:22:12.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:22:12.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:22:12.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:22:12.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:22:12.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:22:12.042 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:22:12.042 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:22:12.042 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:22:12.042 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-13T02:22:12.042 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-13T02:22:12.042 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:22:12.042 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=da87c190-1e2f-415a-9ad3-d9fdcbec303d 2026-02-13T02:22:12.043 INFO:tasks.workunit.client.0.trial112.stdout:add osd2 da87c190-1e2f-415a-9ad3-d9fdcbec303d 2026-02-13T02:22:12.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 da87c190-1e2f-415a-9ad3-d9fdcbec303d' 2026-02-13T02:22:12.043 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:22:12.054 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDUio5p3fSiAhAAnfscMcXRw0NpCRGdbgLJIg== 2026-02-13T02:22:12.054 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDUio5p3fSiAhAAnfscMcXRw0NpCRGdbgLJIg=="}' 2026-02-13T02:22:12.054 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new da87c190-1e2f-415a-9ad3-d9fdcbec303d -i td/osd-scrub-repair/2/new.json 2026-02-13T02:22:12.213 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:22:12.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-13T02:22:12.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDUio5p3fSiAhAAnfscMcXRw0NpCRGdbgLJIg== --osd-uuid da87c190-1e2f-415a-9ad3-d9fdcbec303d 2026-02-13T02:22:12.253 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:12.239+0000 7fe39d564a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:22:12.255 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:12.239+0000 7fe39d564a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:22:12.256 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:12.239+0000 7fe39d564a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:22:12.256 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:12.243+0000 7fe39d564a40 -1 bdev(0x564fac8dac00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-13T02:22:12.257 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:12.243+0000 7fe39d564a40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-13T02:22:12.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-13T02:22:12.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:22:12.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-13T02:22:12.935 INFO:tasks.workunit.client.0.trial112.stdout:adding osd2 key to auth repository 2026-02-13T02:22:12.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:22:13.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-13T02:22:13.139 INFO:tasks.workunit.client.0.trial112.stdout:start osd.2 2026-02-13T02:22:13.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T02:22:13.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:22:13.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:22:13.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:22:13.164 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:13.147+0000 7fd68b703a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:22:13.165 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:13.151+0000 7fd68b703a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:22:13.166 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:13.151+0000 7fd68b703a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:22:13.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-13T02:22:13.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:22:13.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T02:22:13.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:22:13.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:22:13.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:22:13.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:22:13.305 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:22:13.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:22:13.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:22:13.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:22:13.517 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:13.503+0000 7fd68b703a40 -1 Falling back to public interface 2026-02-13T02:22:14.126 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:14.111+0000 7fd68b703a40 -1 osd.2 0 log_to_monitors true 2026-02-13T02:22:14.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:22:14.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:22:14.469 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:22:14.469 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:22:14.469 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:22:14.469 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:22:14.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:22:15.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:22:15.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:22:15.632 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:22:15.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:22:15.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:22:15.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:22:15.789 INFO:tasks.workunit.client.0.trial112.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/702258511,v1:127.0.0.1:6819/702258511] [v2:127.0.0.1:6820/702258511,v1:127.0.0.1:6821/702258511] exists,up da87c190-1e2f-415a-9ad3-d9fdcbec303d 2026-02-13T02:22:15.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:22:15.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:22:15.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:22:15.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:801: TEST_repair_stats_ec: create_ec_pool testpool false k=2 m=1 2026-02-13T02:22:15.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2522: create_ec_pool: local pool_name=testpool 2026-02-13T02:22:15.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2523: create_ec_pool: shift 2026-02-13T02:22:15.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2524: create_ec_pool: local allow_overwrites=false 2026-02-13T02:22:15.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2525: create_ec_pool: shift 2026-02-13T02:22:15.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2527: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2026-02-13T02:22:15.994 INFO:tasks.workunit.client.0.trial112.stderr:load: isa 2026-02-13T02:22:16.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2529: create_ec_pool: create_pool testpool 1 1 erasure myprofile 2026-02-13T02:22:16.006 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create testpool 1 1 erasure myprofile 2026-02-13T02:22:16.297 INFO:tasks.workunit.client.0.trial112.stderr:pool 'testpool' created 2026-02-13T02:22:16.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:22:17.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2531: create_ec_pool: '[' false = true ']' 2026-02-13T02:22:17.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2535: create_ec_pool: wait_for_clean 2026-02-13T02:22:17.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:22:17.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:22:17.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:22:17.313 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:22:17.314 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:22:17.314 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:22:17.315 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:22:17.315 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:22:17.315 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:22:17.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:22:17.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:22:17.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:22:17.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:22:17.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:22:17.437 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:22:17.597 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:22:17.597 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:22:17.598 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:22:17.598 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:22:17.598 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:22:17.598 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:22:17.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-13T02:22:17.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-13T02:22:17.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-13T02:22:17.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:22:17.671 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:22:17.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705667 2026-02-13T02:22:17.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705667 2026-02-13T02:22:17.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705667' 2026-02-13T02:22:17.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:22:17.746 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:22:17.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542146 2026-02-13T02:22:17.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542146 2026-02-13T02:22:17.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705667 2-60129542146' 2026-02-13T02:22:17.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:22:17.819 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-13T02:22:17.819 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:22:17.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:22:17.821 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-13T02:22:17.821 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:22:17.823 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-13T02:22:17.823 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836483 2026-02-13T02:22:17.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-13T02:22:17.824 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:22:17.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-13T02:22:17.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:22:18.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:22:18.998 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:22:19.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-13T02:22:19.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:22:20.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:22:20.166 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:22:20.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836483 2026-02-13T02:22:20.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:22:20.326 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705667 2026-02-13T02:22:20.326 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:22:20.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:22:20.329 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705667 2026-02-13T02:22:20.329 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:22:20.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705667 2026-02-13T02:22:20.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705667' 2026-02-13T02:22:20.330 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705667 2026-02-13T02:22:20.331 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:22:20.506 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705667 -lt 38654705667 2026-02-13T02:22:20.506 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:22:20.507 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542146 2026-02-13T02:22:20.507 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:22:20.508 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:22:20.509 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542146 2026-02-13T02:22:20.509 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:22:20.511 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542146 2026-02-13T02:22:20.511 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542146 2026-02-13T02:22:20.511 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542146' 2026-02-13T02:22:20.512 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:22:20.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542146 -lt 60129542146 2026-02-13T02:22:20.683 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:22:20.684 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:22:20.684 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:22:20.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:22:20.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:22:20.884 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:22:20.884 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:22:20.884 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:22:20.884 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:22:20.885 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:22:20.885 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:22:21.050 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:22:21.051 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:22:21.051 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:22:21.051 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:22:21.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:22:21.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:22:21.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:22:21.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2536: create_ec_pool: return 0 2026-02-13T02:22:21.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: TEST_repair_stats_ec: local payload=ABCDEF 2026-02-13T02:22:21.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: TEST_repair_stats_ec: echo ABCDEF 2026-02-13T02:22:21.245 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: seq 1 30 2026-02-13T02:22:21.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-13T02:22:21.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:22:21.266 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-13T02:22:21.266 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:22:21.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-13T02:22:21.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:22:21.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-13T02:22:21.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:22:21.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-13T02:22:21.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:22:21.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-13T02:22:21.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:22:21.381 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-13T02:22:21.381 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:22:21.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-13T02:22:21.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:22:21.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-13T02:22:21.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:22:21.456 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-13T02:22:21.456 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:22:21.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-13T02:22:21.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj11 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:22:21.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-13T02:22:21.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj12 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:22:21.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-13T02:22:21.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj13 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:22:21.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-13T02:22:21.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj14 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:22:21.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-13T02:22:21.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj15 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:22:21.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-13T02:22:21.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj16 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:22:21.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-13T02:22:21.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj17 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:22:21.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-13T02:22:21.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj18 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:22:21.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-13T02:22:21.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj19 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:22:21.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-13T02:22:21.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj20 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:22:21.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-13T02:22:21.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj21 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:22:21.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-13T02:22:21.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj22 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:22:21.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-13T02:22:21.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj23 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:22:21.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-13T02:22:21.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj24 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:22:21.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-13T02:22:21.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj25 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:22:21.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-13T02:22:21.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj26 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:22:21.854 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-13T02:22:21.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj27 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:22:21.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-13T02:22:21.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj28 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:22:21.906 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-13T02:22:21.906 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj29 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:22:21.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-13T02:22:21.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj30 td/osd-scrub-repair/ORIGINAL 2026-02-13T02:22:21.952 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:813: TEST_repair_stats_ec: get_not_primary testpool obj1 2026-02-13T02:22:21.952 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-13T02:22:21.952 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=obj1 2026-02-13T02:22:21.953 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool obj1 2026-02-13T02:22:21.953 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-13T02:22:21.953 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-13T02:22:21.953 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2026-02-13T02:22:21.953 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T02:22:22.115 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-13T02:22:22.116 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool obj1 2026-02-13T02:22:22.116 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-13T02:22:22.277 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:813: TEST_repair_stats_ec: local other=0 2026-02-13T02:22:22.277 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:814: TEST_repair_stats_ec: get_pg testpool obj1 2026-02-13T02:22:22.278 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2026-02-13T02:22:22.278 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=obj1 2026-02-13T02:22:22.278 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool obj1 2026-02-13T02:22:22.278 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T02:22:22.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:814: TEST_repair_stats_ec: local pgid=1.0 2026-02-13T02:22:22.437 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:815: TEST_repair_stats_ec: get_primary testpool obj1 2026-02-13T02:22:22.437 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-13T02:22:22.437 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-13T02:22:22.437 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2026-02-13T02:22:22.437 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T02:22:22.598 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:815: TEST_repair_stats_ec: local primary=1 2026-02-13T02:22:22.598 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:817: TEST_repair_stats_ec: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T02:22:22.599 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:22:22.599 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:22:22.600 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:22:22.600 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:22:22.600 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:22:22.908 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:22:22.908 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:818: TEST_repair_stats_ec: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-13T02:22:22.909 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:22:22.909 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:22:22.910 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:22:22.910 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:22:22.910 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:22:23.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:22:23.221 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: seq 1 26 2026-02-13T02:22:23.223 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-13T02:22:23.223 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 1 % 2 2026-02-13T02:22:23.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=1 2026-02-13T02:22:23.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 remove 2026-02-13T02:22:23.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:22:23.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:22:23.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:22:23.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:22:23.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:22:23.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 remove 2026-02-13T02:22:23.432 INFO:tasks.workunit.client.0.trial112.stdout:remove 0#1:ff7b1f36:::obj1:head# 2026-02-13T02:22:23.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-13T02:22:23.520 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 2 % 2 2026-02-13T02:22:23.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=0 2026-02-13T02:22:23.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2026-02-13T02:22:23.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:22:23.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:22:23.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:22:23.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:22:23.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:22:23.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2026-02-13T02:22:23.752 INFO:tasks.workunit.client.0.trial112.stdout:remove 1#1:104778fc:::obj2:head# 2026-02-13T02:22:23.848 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-13T02:22:23.848 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 3 % 2 2026-02-13T02:22:23.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=1 2026-02-13T02:22:23.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj3 remove 2026-02-13T02:22:23.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:22:23.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:22:23.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:22:23.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:22:23.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:22:23.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj3 remove 2026-02-13T02:22:24.029 INFO:tasks.workunit.client.0.trial112.stdout:remove 0#1:8dd16f86:::obj3:head# 2026-02-13T02:22:24.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-13T02:22:24.136 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 4 % 2 2026-02-13T02:22:24.138 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=0 2026-02-13T02:22:24.138 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj4 remove 2026-02-13T02:22:24.138 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:22:24.138 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:22:24.138 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:22:24.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:22:24.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:22:24.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj4 remove 2026-02-13T02:22:24.314 INFO:tasks.workunit.client.0.trial112.stdout:remove 1#1:0ee9ae15:::obj4:head# 2026-02-13T02:22:24.435 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-13T02:22:24.436 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 5 % 2 2026-02-13T02:22:24.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=1 2026-02-13T02:22:24.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj5 remove 2026-02-13T02:22:24.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:22:24.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:22:24.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:22:24.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:22:24.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:22:24.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj5 remove 2026-02-13T02:22:24.638 INFO:tasks.workunit.client.0.trial112.stdout:remove 0#1:c52c9666:::obj5:head# 2026-02-13T02:22:24.739 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-13T02:22:24.740 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 6 % 2 2026-02-13T02:22:24.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=0 2026-02-13T02:22:24.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj6 remove 2026-02-13T02:22:24.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:22:24.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:22:24.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:22:24.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:22:24.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:22:24.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj6 remove 2026-02-13T02:22:24.899 INFO:tasks.workunit.client.0.trial112.stdout:remove 1#1:233a42c1:::obj6:head# 2026-02-13T02:22:24.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-13T02:22:25.000 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 7 % 2 2026-02-13T02:22:25.001 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=1 2026-02-13T02:22:25.001 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj7 remove 2026-02-13T02:22:25.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:22:25.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:22:25.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:22:25.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:22:25.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:22:25.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj7 remove 2026-02-13T02:22:25.186 INFO:tasks.workunit.client.0.trial112.stdout:remove 0#1:9a09113a:::obj7:head# 2026-02-13T02:22:25.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-13T02:22:25.304 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 8 % 2 2026-02-13T02:22:25.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=0 2026-02-13T02:22:25.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj8 remove 2026-02-13T02:22:25.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:22:25.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:22:25.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:22:25.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:22:25.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:22:25.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj8 remove 2026-02-13T02:22:25.482 INFO:tasks.workunit.client.0.trial112.stdout:remove 1#1:a557efb1:::obj8:head# 2026-02-13T02:22:25.575 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-13T02:22:25.576 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 9 % 2 2026-02-13T02:22:25.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=1 2026-02-13T02:22:25.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj9 remove 2026-02-13T02:22:25.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:22:25.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:22:25.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:22:25.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:22:25.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:22:25.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj9 remove 2026-02-13T02:22:25.763 INFO:tasks.workunit.client.0.trial112.stdout:remove 0#1:fe351e27:::obj9:head# 2026-02-13T02:22:25.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-13T02:22:25.864 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 10 % 2 2026-02-13T02:22:25.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=0 2026-02-13T02:22:25.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj10 remove 2026-02-13T02:22:25.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:22:25.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:22:25.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:22:25.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:22:25.866 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:22:25.866 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj10 remove 2026-02-13T02:22:26.055 INFO:tasks.workunit.client.0.trial112.stdout:remove 1#1:d1337354:::obj10:head# 2026-02-13T02:22:26.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-13T02:22:26.140 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 11 % 2 2026-02-13T02:22:26.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=1 2026-02-13T02:22:26.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj11 remove 2026-02-13T02:22:26.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:22:26.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:22:26.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:22:26.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:22:26.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:22:26.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj11 remove 2026-02-13T02:22:26.314 INFO:tasks.workunit.client.0.trial112.stdout:remove 0#1:eeae2a94:::obj11:head# 2026-02-13T02:22:26.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-13T02:22:26.409 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 12 % 2 2026-02-13T02:22:26.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=0 2026-02-13T02:22:26.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj12 remove 2026-02-13T02:22:26.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:22:26.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:22:26.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:22:26.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:22:26.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:22:26.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj12 remove 2026-02-13T02:22:26.579 INFO:tasks.workunit.client.0.trial112.stdout:remove 1#1:af5f95cb:::obj12:head# 2026-02-13T02:22:26.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-13T02:22:26.668 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 13 % 2 2026-02-13T02:22:26.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=1 2026-02-13T02:22:26.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj13 remove 2026-02-13T02:22:26.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:22:26.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:22:26.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:22:26.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:22:26.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:22:26.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj13 remove 2026-02-13T02:22:26.839 INFO:tasks.workunit.client.0.trial112.stdout:remove 0#1:a61ad63e:::obj13:head# 2026-02-13T02:22:26.929 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-13T02:22:26.930 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 14 % 2 2026-02-13T02:22:26.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=0 2026-02-13T02:22:26.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj14 remove 2026-02-13T02:22:26.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:22:26.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:22:26.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:22:26.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:22:26.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:22:26.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj14 remove 2026-02-13T02:22:27.110 INFO:tasks.workunit.client.0.trial112.stdout:remove 1#1:e0a44829:::obj14:head# 2026-02-13T02:22:27.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-13T02:22:27.212 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 15 % 2 2026-02-13T02:22:27.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=1 2026-02-13T02:22:27.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj15 remove 2026-02-13T02:22:27.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:22:27.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:22:27.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:22:27.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:22:27.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:22:27.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj15 remove 2026-02-13T02:22:27.383 INFO:tasks.workunit.client.0.trial112.stdout:remove 0#1:ab946124:::obj15:head# 2026-02-13T02:22:27.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-13T02:22:27.480 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 16 % 2 2026-02-13T02:22:27.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=0 2026-02-13T02:22:27.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj16 remove 2026-02-13T02:22:27.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:22:27.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:22:27.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:22:27.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:22:27.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:22:27.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj16 remove 2026-02-13T02:22:27.667 INFO:tasks.workunit.client.0.trial112.stdout:remove 1#1:461f8b5e:::obj16:head# 2026-02-13T02:22:27.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-13T02:22:27.772 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 17 % 2 2026-02-13T02:22:27.774 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=1 2026-02-13T02:22:27.774 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj17 remove 2026-02-13T02:22:27.774 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:22:27.774 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:22:27.775 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:22:27.775 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:22:27.775 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:22:27.775 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj17 remove 2026-02-13T02:22:27.951 INFO:tasks.workunit.client.0.trial112.stdout:remove 0#1:24fc6e92:::obj17:head# 2026-02-13T02:22:28.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-13T02:22:28.032 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 18 % 2 2026-02-13T02:22:28.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=0 2026-02-13T02:22:28.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj18 remove 2026-02-13T02:22:28.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:22:28.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:22:28.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:22:28.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:22:28.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:22:28.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj18 remove 2026-02-13T02:22:28.207 INFO:tasks.workunit.client.0.trial112.stdout:remove 1#1:b9836a99:::obj18:head# 2026-02-13T02:22:28.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-13T02:22:28.312 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 19 % 2 2026-02-13T02:22:28.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=1 2026-02-13T02:22:28.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj19 remove 2026-02-13T02:22:28.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:22:28.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:22:28.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:22:28.315 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:22:28.315 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:22:28.315 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj19 remove 2026-02-13T02:22:28.479 INFO:tasks.workunit.client.0.trial112.stdout:remove 0#1:00b31b6c:::obj19:head# 2026-02-13T02:22:28.575 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-13T02:22:28.576 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 20 % 2 2026-02-13T02:22:28.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=0 2026-02-13T02:22:28.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj20 remove 2026-02-13T02:22:28.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:22:28.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:22:28.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:22:28.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:22:28.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:22:28.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj20 remove 2026-02-13T02:22:28.767 INFO:tasks.workunit.client.0.trial112.stdout:remove 1#1:24b5611b:::obj20:head# 2026-02-13T02:22:28.852 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-13T02:22:28.852 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 21 % 2 2026-02-13T02:22:28.854 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=1 2026-02-13T02:22:28.854 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj21 remove 2026-02-13T02:22:28.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:22:28.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:22:28.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:22:28.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:22:28.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:22:28.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj21 remove 2026-02-13T02:22:29.010 INFO:tasks.workunit.client.0.trial112.stdout:remove 0#1:1243fe1a:::obj21:head# 2026-02-13T02:22:29.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-13T02:22:29.112 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 22 % 2 2026-02-13T02:22:29.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=0 2026-02-13T02:22:29.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj22 remove 2026-02-13T02:22:29.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:22:29.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:22:29.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:22:29.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:22:29.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:22:29.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj22 remove 2026-02-13T02:22:29.291 INFO:tasks.workunit.client.0.trial112.stdout:remove 1#1:c0c49def:::obj22:head# 2026-02-13T02:22:29.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-13T02:22:29.420 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 23 % 2 2026-02-13T02:22:29.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=1 2026-02-13T02:22:29.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj23 remove 2026-02-13T02:22:29.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:22:29.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:22:29.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:22:29.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:22:29.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:22:29.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj23 remove 2026-02-13T02:22:29.594 INFO:tasks.workunit.client.0.trial112.stdout:remove 0#1:242c40e0:::obj23:head# 2026-02-13T02:22:29.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-13T02:22:29.704 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 24 % 2 2026-02-13T02:22:29.705 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=0 2026-02-13T02:22:29.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj24 remove 2026-02-13T02:22:29.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:22:29.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:22:29.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:22:29.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:22:29.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:22:29.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj24 remove 2026-02-13T02:22:29.906 INFO:tasks.workunit.client.0.trial112.stdout:remove 1#1:7eacede5:::obj24:head# 2026-02-13T02:22:29.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-13T02:22:30.000 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 25 % 2 2026-02-13T02:22:30.001 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=1 2026-02-13T02:22:30.001 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj25 remove 2026-02-13T02:22:30.001 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:22:30.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:22:30.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:22:30.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:22:30.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:22:30.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj25 remove 2026-02-13T02:22:30.198 INFO:tasks.workunit.client.0.trial112.stdout:remove 0#1:7109478f:::obj25:head# 2026-02-13T02:22:30.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-13T02:22:30.308 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 26 % 2 2026-02-13T02:22:30.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=0 2026-02-13T02:22:30.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj26 remove 2026-02-13T02:22:30.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:22:30.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:22:30.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:22:30.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:22:30.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:22:30.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj26 remove 2026-02-13T02:22:30.499 INFO:tasks.workunit.client.0.trial112.stdout:remove 1#1:f3c2593a:::obj26:head# 2026-02-13T02:22:30.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:825: TEST_repair_stats_ec: activate_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T02:22:30.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:22:30.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:22:30.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:22:30.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:22:30.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:22:30.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:22:30.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:22:30.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:22:30.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:22:30.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:22:30.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:22:30.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:22:30.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:22:30.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:22:30.612 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:22:30.612 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:22:30.612 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:22:30.613 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:22:30.613 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:22:30.613 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:22:30.613 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:22:30.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:22:30.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:22:30.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:22:30.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:22:30.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:22:30.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:22:30.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:22:30.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:22:30.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:22:30.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-13T02:22:30.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:22:30.615 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T02:22:30.615 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:22:30.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T02:22:30.616 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-13T02:22:30.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:22:30.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:22:30.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:22:30.618 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:22:30.635 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:30.623+0000 7fce51cf4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:22:30.637 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:30.623+0000 7fce51cf4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:22:30.638 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:30.623+0000 7fce51cf4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:22:30.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:22:30.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:22:30.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:22:30.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:22:30.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:22:30.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:22:30.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:22:30.781 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:22:30.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:22:30.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:22:30.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:22:31.033 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:31.019+0000 7fce51cf4a40 -1 Falling back to public interface 2026-02-13T02:22:31.663 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:31.647+0000 7fce51cf4a40 -1 osd.1 22 log_to_monitors true 2026-02-13T02:22:31.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:22:31.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:22:31.949 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:22:31.950 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:22:31.950 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:22:31.950 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:22:32.111 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 26 up_thru 26 down_at 23 last_clean_interval [9,22) [v2:127.0.0.1:6800/2196397484,v1:127.0.0.1:6801/2196397484] [v2:127.0.0.1:6802/2196397484,v1:127.0.0.1:6803/2196397484] exists,up 5a27e272-65c7-49e4-a7e7-27bfe069e85a 2026-02-13T02:22:32.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:22:32.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:22:32.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:22:32.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:826: TEST_repair_stats_ec: activate_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T02:22:32.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:22:32.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:22:32.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:22:32.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:22:32.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:22:32.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:22:32.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:22:32.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:22:32.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:22:32.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:22:32.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:22:32.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:22:32.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:22:32.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:22:32.116 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:22:32.116 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:22:32.116 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:22:32.116 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:22:32.116 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:22:32.116 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:22:32.117 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:22:32.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:22:32.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:22:32.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:22:32.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:22:32.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:22:32.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:22:32.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:22:32.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:22:32.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:22:32.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-13T02:22:32.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:22:32.118 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T02:22:32.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:22:32.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-13T02:22:32.119 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T02:22:32.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:22:32.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:22:32.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:22:32.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:22:32.133 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:32.119+0000 7f817c985a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:22:32.135 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:32.119+0000 7f817c985a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:22:32.136 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:32.123+0000 7f817c985a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:22:32.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:22:32.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:22:32.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:22:32.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:22:32.280 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:22:32.280 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:22:32.280 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:22:32.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:22:32.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:22:32.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:22:32.439 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:22:32.509 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:32.495+0000 7f817c985a40 -1 Falling back to public interface 2026-02-13T02:22:33.126 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:33.111+0000 7f817c985a40 -1 osd.0 21 log_to_monitors true 2026-02-13T02:22:33.441 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:22:33.441 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:22:33.441 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:22:33.442 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:22:33.442 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:22:33.442 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:22:33.603 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 29 up_thru 0 down_at 21 last_clean_interval [5,21) [v2:127.0.0.1:6810/3988728760,v1:127.0.0.1:6811/3988728760] [v2:127.0.0.1:6812/3988728760,v1:127.0.0.1:6813/3988728760] exists,up a2bceff7-8c06-46ae-bc7a-5b0e7f1db95e 2026-02-13T02:22:33.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:22:33.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:22:33.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:22:33.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:827: TEST_repair_stats_ec: wait_for_clean 2026-02-13T02:22:33.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:22:33.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:22:33.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:22:33.605 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:22:33.605 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:22:33.606 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:22:33.606 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:22:33.606 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:22:33.606 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:22:33.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:22:33.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:22:33.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:22:33.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:22:33.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:22:33.725 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:22:33.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:22:33.889 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:22:33.889 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:22:33.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:22:33.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:22:33.889 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:22:33.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=124554051586 2026-02-13T02:22:33.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 124554051586 2026-02-13T02:22:33.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-124554051586' 2026-02-13T02:22:33.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:22:33.962 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:22:34.037 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=111669149698 2026-02-13T02:22:34.037 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 111669149698 2026-02-13T02:22:34.037 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-124554051586 1-111669149698' 2026-02-13T02:22:34.037 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:22:34.038 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:22:34.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542150 2026-02-13T02:22:34.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542150 2026-02-13T02:22:34.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-124554051586 1-111669149698 2-60129542150' 2026-02-13T02:22:34.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:22:34.114 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-124554051586 2026-02-13T02:22:34.114 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:22:34.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:22:34.116 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-124554051586 2026-02-13T02:22:34.117 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:22:34.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=124554051586 2026-02-13T02:22:34.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 124554051586' 2026-02-13T02:22:34.119 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 124554051586 2026-02-13T02:22:34.119 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:22:34.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 124554051586 2026-02-13T02:22:34.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:22:35.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:22:35.289 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:22:35.456 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 124554051586 -lt 124554051586 2026-02-13T02:22:35.456 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:22:35.457 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-111669149698 2026-02-13T02:22:35.457 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:22:35.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:22:35.460 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-111669149698 2026-02-13T02:22:35.460 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:22:35.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=111669149698 2026-02-13T02:22:35.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 111669149698' 2026-02-13T02:22:35.462 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 111669149698 2026-02-13T02:22:35.462 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:22:35.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 111669149698 -lt 111669149698 2026-02-13T02:22:35.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:22:35.631 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542150 2026-02-13T02:22:35.631 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:22:35.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:22:35.633 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542150 2026-02-13T02:22:35.633 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:22:35.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542150 2026-02-13T02:22:35.636 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542150 2026-02-13T02:22:35.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542150' 2026-02-13T02:22:35.636 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:22:35.801 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542150 -lt 60129542150 2026-02-13T02:22:35.802 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:22:35.803 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:22:35.803 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:22:36.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:22:36.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:22:36.004 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:22:36.004 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:22:36.004 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:22:36.004 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:22:36.004 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:22:36.005 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:22:36.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:22:36.169 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:22:36.169 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:22:36.169 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:22:36.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:22:36.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:22:36.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:22:36.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:829: TEST_repair_stats_ec: repair 1.0 2026-02-13T02:22:36.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=1.0 2026-02-13T02:22:36.363 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 1.0 2026-02-13T02:22:36.363 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:22:36.363 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:22:36.364 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:22:36.364 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:22:36.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-13T02:22:16.287207+0000 2026-02-13T02:22:36.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 1.0 2026-02-13T02:22:36.674 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 1.0s0 on osd.1 to repair 2026-02-13T02:22:36.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 1.0 2026-02-13T02:22:16.287207+0000 2026-02-13T02:22:36.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-13T02:22:36.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:22:16.287207+0000 2026-02-13T02:22:36.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:22:36.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:22:36.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:22:36.685 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:22:36.685 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:22:36.685 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:22:36.685 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:22:36.686 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:22:36.845 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:22:16.287207+0000 '>' 2026-02-13T02:22:16.287207+0000 2026-02-13T02:22:36.845 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:22:37.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:22:37.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:22:37.847 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:22:37.847 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:22:37.847 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:22:37.848 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:22:37.848 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:22:38.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:22:16.287207+0000 '>' 2026-02-13T02:22:16.287207+0000 2026-02-13T02:22:38.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:22:39.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:22:39.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:22:39.011 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:22:39.011 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:22:39.011 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:22:39.011 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:22:39.011 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:22:39.167 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:22:16.287207+0000 '>' 2026-02-13T02:22:16.287207+0000 2026-02-13T02:22:39.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:22:40.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:22:40.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:22:40.169 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:22:40.169 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:22:40.169 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:22:40.170 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:22:40.170 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:22:40.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:22:16.287207+0000 '>' 2026-02-13T02:22:16.287207+0000 2026-02-13T02:22:40.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:22:41.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:22:41.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:22:41.329 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:22:41.329 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:22:41.330 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:22:41.330 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:22:41.330 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:22:41.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:22:16.287207+0000 '>' 2026-02-13T02:22:16.287207+0000 2026-02-13T02:22:41.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:22:42.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:22:42.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:22:42.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:22:42.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:22:42.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:22:42.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:22:42.489 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:22:42.645 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:22:16.287207+0000 '>' 2026-02-13T02:22:16.287207+0000 2026-02-13T02:22:42.646 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:22:43.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:22:43.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:22:43.648 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:22:43.648 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:22:43.648 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:22:43.648 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:22:43.649 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:22:43.806 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:22:37.651842+0000 '>' 2026-02-13T02:22:16.287207+0000 2026-02-13T02:22:43.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:22:43.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:830: TEST_repair_stats_ec: wait_for_clean 2026-02-13T02:22:43.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:22:43.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:22:43.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:22:43.808 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:22:43.808 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:22:43.808 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:22:43.808 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:22:43.808 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:22:43.808 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:22:43.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:22:43.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:22:43.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:22:43.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:22:43.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:22:43.931 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:22:44.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:22:44.100 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:22:44.100 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:22:44.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:22:44.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:22:44.101 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:22:44.175 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=124554051589 2026-02-13T02:22:44.175 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 124554051589 2026-02-13T02:22:44.175 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-124554051589' 2026-02-13T02:22:44.175 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:22:44.175 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:22:44.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=111669149701 2026-02-13T02:22:44.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 111669149701 2026-02-13T02:22:44.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-124554051589 1-111669149701' 2026-02-13T02:22:44.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:22:44.249 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:22:44.319 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542153 2026-02-13T02:22:44.319 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542153 2026-02-13T02:22:44.319 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-124554051589 1-111669149701 2-60129542153' 2026-02-13T02:22:44.319 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:22:44.320 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-124554051589 2026-02-13T02:22:44.320 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:22:44.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:22:44.322 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-124554051589 2026-02-13T02:22:44.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:22:44.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=124554051589 2026-02-13T02:22:44.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 124554051589' 2026-02-13T02:22:44.324 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 124554051589 2026-02-13T02:22:44.325 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:22:44.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 124554051587 -lt 124554051589 2026-02-13T02:22:44.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:22:45.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:22:45.491 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:22:45.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 124554051589 -lt 124554051589 2026-02-13T02:22:45.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:22:45.654 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-111669149701 2026-02-13T02:22:45.655 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:22:45.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:22:45.657 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-111669149701 2026-02-13T02:22:45.657 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:22:45.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=111669149701 2026-02-13T02:22:45.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 111669149701' 2026-02-13T02:22:45.659 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 111669149701 2026-02-13T02:22:45.660 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:22:45.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 111669149701 -lt 111669149701 2026-02-13T02:22:45.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:22:45.826 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542153 2026-02-13T02:22:45.826 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:22:45.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:22:45.829 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542153 2026-02-13T02:22:45.829 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:22:45.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542153 2026-02-13T02:22:45.831 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542153 2026-02-13T02:22:45.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542153' 2026-02-13T02:22:45.831 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:22:46.004 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542153 -lt 60129542153 2026-02-13T02:22:46.005 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:22:46.005 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:22:46.005 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:22:46.206 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:22:46.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:22:46.207 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:22:46.208 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:22:46.208 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:22:46.208 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:22:46.208 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:22:46.208 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:22:46.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:22:46.368 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:22:46.368 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:22:46.369 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:22:46.557 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:22:46.557 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:22:46.557 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:22:46.557 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: TEST_repair_stats_ec: ceph pg dump pgs 2026-02-13T02:22:46.708 INFO:tasks.workunit.client.0.trial112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-13T02:22:46.708 INFO:tasks.workunit.client.0.trial112.stdout:1.0 30 0 0 0 0 210 0 0 30 0 30 active+clean 2026-02-13T02:22:37.686613+0000 20'30 31:140 [1,0,2] 1 [1,0,2] 1 20'30 2026-02-13T02:22:37.651842+0000 20'30 2026-02-13T02:22:37.651842+0000 0 1 periodic scrub scheduled @ 2026-02-14T02:22:37.651842+0000 17 0 2026-02-13T02:22:46.708 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T02:22:46.708 INFO:tasks.workunit.client.0.trial112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-13T02:22:46.708 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:22:46.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: TEST_repair_stats_ec: flush_pg_stats 2026-02-13T02:22:46.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:22:46.717 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:22:46.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:22:46.895 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:22:46.895 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:22:46.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:22:46.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:22:46.895 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:22:46.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=124554051590 2026-02-13T02:22:46.970 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 124554051590 2026-02-13T02:22:46.970 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-124554051590' 2026-02-13T02:22:46.970 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:22:46.970 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:22:47.042 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=111669149703 2026-02-13T02:22:47.042 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 111669149703 2026-02-13T02:22:47.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-124554051590 1-111669149703' 2026-02-13T02:22:47.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:22:47.043 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:22:47.121 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542155 2026-02-13T02:22:47.121 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542155 2026-02-13T02:22:47.121 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-124554051590 1-111669149703 2-60129542155' 2026-02-13T02:22:47.121 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:22:47.122 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-124554051590 2026-02-13T02:22:47.122 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:22:47.123 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:22:47.124 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-124554051590 2026-02-13T02:22:47.125 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:22:47.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=124554051590 2026-02-13T02:22:47.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 124554051590' 2026-02-13T02:22:47.127 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 124554051590 2026-02-13T02:22:47.127 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:22:47.291 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 124554051590 -lt 124554051590 2026-02-13T02:22:47.291 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:22:47.292 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-111669149703 2026-02-13T02:22:47.292 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:22:47.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:22:47.295 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-111669149703 2026-02-13T02:22:47.295 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:22:47.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=111669149703 2026-02-13T02:22:47.297 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 111669149703 2026-02-13T02:22:47.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 111669149703' 2026-02-13T02:22:47.297 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:22:47.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 111669149703 -lt 111669149703 2026-02-13T02:22:47.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:22:47.460 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542155 2026-02-13T02:22:47.460 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:22:47.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:22:47.462 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542155 2026-02-13T02:22:47.462 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:22:47.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542155 2026-02-13T02:22:47.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542155' 2026-02-13T02:22:47.464 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542155 2026-02-13T02:22:47.465 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:22:47.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542155 -lt 60129542155 2026-02-13T02:22:47.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:835: TEST_repair_stats_ec: ceph pg 1.0 query 2026-02-13T02:22:47.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:835: TEST_repair_stats_ec: jq .info.stats.stat_sum 2026-02-13T02:22:47.703 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:22:47.704 INFO:tasks.workunit.client.0.trial112.stdout: "num_bytes": 210, 2026-02-13T02:22:47.704 INFO:tasks.workunit.client.0.trial112.stdout: "num_objects": 30, 2026-02-13T02:22:47.704 INFO:tasks.workunit.client.0.trial112.stdout: "num_object_clones": 0, 2026-02-13T02:22:47.704 INFO:tasks.workunit.client.0.trial112.stdout: "num_object_copies": 90, 2026-02-13T02:22:47.704 INFO:tasks.workunit.client.0.trial112.stdout: "num_objects_missing_on_primary": 0, 2026-02-13T02:22:47.704 INFO:tasks.workunit.client.0.trial112.stdout: "num_objects_missing": 0, 2026-02-13T02:22:47.704 INFO:tasks.workunit.client.0.trial112.stdout: "num_objects_degraded": 0, 2026-02-13T02:22:47.704 INFO:tasks.workunit.client.0.trial112.stdout: "num_objects_misplaced": 0, 2026-02-13T02:22:47.704 INFO:tasks.workunit.client.0.trial112.stdout: "num_objects_unfound": 0, 2026-02-13T02:22:47.704 INFO:tasks.workunit.client.0.trial112.stdout: "num_objects_dirty": 30, 2026-02-13T02:22:47.704 INFO:tasks.workunit.client.0.trial112.stdout: "num_whiteouts": 0, 2026-02-13T02:22:47.705 INFO:tasks.workunit.client.0.trial112.stdout: "num_read": 0, 2026-02-13T02:22:47.705 INFO:tasks.workunit.client.0.trial112.stdout: "num_read_kb": 0, 2026-02-13T02:22:47.705 INFO:tasks.workunit.client.0.trial112.stdout: "num_write": 30, 2026-02-13T02:22:47.705 INFO:tasks.workunit.client.0.trial112.stdout: "num_write_kb": 30, 2026-02-13T02:22:47.705 INFO:tasks.workunit.client.0.trial112.stdout: "num_scrub_errors": 0, 2026-02-13T02:22:47.705 INFO:tasks.workunit.client.0.trial112.stdout: "num_shallow_scrub_errors": 0, 2026-02-13T02:22:47.705 INFO:tasks.workunit.client.0.trial112.stdout: "num_deep_scrub_errors": 0, 2026-02-13T02:22:47.705 INFO:tasks.workunit.client.0.trial112.stdout: "num_objects_recovered": 26, 2026-02-13T02:22:47.705 INFO:tasks.workunit.client.0.trial112.stdout: "num_bytes_recovered": 182, 2026-02-13T02:22:47.705 INFO:tasks.workunit.client.0.trial112.stdout: "num_keys_recovered": 0, 2026-02-13T02:22:47.705 INFO:tasks.workunit.client.0.trial112.stdout: "num_objects_omap": 0, 2026-02-13T02:22:47.705 INFO:tasks.workunit.client.0.trial112.stdout: "num_objects_hit_set_archive": 0, 2026-02-13T02:22:47.706 INFO:tasks.workunit.client.0.trial112.stdout: "num_bytes_hit_set_archive": 0, 2026-02-13T02:22:47.706 INFO:tasks.workunit.client.0.trial112.stdout: "num_flush": 0, 2026-02-13T02:22:47.706 INFO:tasks.workunit.client.0.trial112.stdout: "num_flush_kb": 0, 2026-02-13T02:22:47.706 INFO:tasks.workunit.client.0.trial112.stdout: "num_evict": 0, 2026-02-13T02:22:47.706 INFO:tasks.workunit.client.0.trial112.stdout: "num_evict_kb": 0, 2026-02-13T02:22:47.706 INFO:tasks.workunit.client.0.trial112.stdout: "num_promote": 0, 2026-02-13T02:22:47.706 INFO:tasks.workunit.client.0.trial112.stdout: "num_flush_mode_high": 0, 2026-02-13T02:22:47.706 INFO:tasks.workunit.client.0.trial112.stdout: "num_flush_mode_low": 0, 2026-02-13T02:22:47.706 INFO:tasks.workunit.client.0.trial112.stdout: "num_evict_mode_some": 0, 2026-02-13T02:22:47.706 INFO:tasks.workunit.client.0.trial112.stdout: "num_evict_mode_full": 0, 2026-02-13T02:22:47.706 INFO:tasks.workunit.client.0.trial112.stdout: "num_objects_pinned": 0, 2026-02-13T02:22:47.706 INFO:tasks.workunit.client.0.trial112.stdout: "num_legacy_snapsets": 0, 2026-02-13T02:22:47.706 INFO:tasks.workunit.client.0.trial112.stdout: "num_large_omap_objects": 0, 2026-02-13T02:22:47.707 INFO:tasks.workunit.client.0.trial112.stdout: "num_objects_manifest": 0, 2026-02-13T02:22:47.707 INFO:tasks.workunit.client.0.trial112.stdout: "num_omap_bytes": 0, 2026-02-13T02:22:47.707 INFO:tasks.workunit.client.0.trial112.stdout: "num_omap_keys": 0, 2026-02-13T02:22:47.707 INFO:tasks.workunit.client.0.trial112.stdout: "num_objects_repaired": 26 2026-02-13T02:22:47.707 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:22:47.707 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: TEST_repair_stats_ec: ceph pg 1.0 query 2026-02-13T02:22:47.708 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: TEST_repair_stats_ec: jq .info.stats.stat_sum.num_objects_repaired 2026-02-13T02:22:47.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: TEST_repair_stats_ec: COUNT=26 2026-02-13T02:22:47.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: TEST_repair_stats_ec: test 26 = 26 2026-02-13T02:22:47.782 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:839: TEST_repair_stats_ec: expr 3 - 1 2026-02-13T02:22:47.784 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:839: TEST_repair_stats_ec: seq 0 2 2026-02-13T02:22:47.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:839: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:22:47.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:840: TEST_repair_stats_ec: '[' 0 = 0 -o 0 = 1 ']' 2026-02-13T02:22:47.785 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:841: TEST_repair_stats_ec: expr 26 / 2 2026-02-13T02:22:47.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:841: TEST_repair_stats_ec: repair=13 2026-02-13T02:22:47.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:846: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2026-02-13T02:22:47.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:846: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 0 )' 2026-02-13T02:22:47.932 INFO:tasks.workunit.client.0.trial112.stderr:dumped all 2026-02-13T02:22:47.942 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:22:47.943 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 0, 2026-02-13T02:22:47.943 INFO:tasks.workunit.client.0.trial112.stdout: "up_from": 29, 2026-02-13T02:22:47.943 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 124554051590, 2026-02-13T02:22:47.943 INFO:tasks.workunit.client.0.trial112.stdout: "num_pgs": 1, 2026-02-13T02:22:47.943 INFO:tasks.workunit.client.0.trial112.stdout: "num_osds": 1, 2026-02-13T02:22:47.943 INFO:tasks.workunit.client.0.trial112.stdout: "num_per_pool_osds": 1, 2026-02-13T02:22:47.943 INFO:tasks.workunit.client.0.trial112.stdout: "num_per_pool_omap_osds": 1, 2026-02-13T02:22:47.943 INFO:tasks.workunit.client.0.trial112.stdout: "kb": 104857600, 2026-02-13T02:22:47.943 INFO:tasks.workunit.client.0.trial112.stdout: "kb_used": 28488, 2026-02-13T02:22:47.943 INFO:tasks.workunit.client.0.trial112.stdout: "kb_used_data": 376, 2026-02-13T02:22:47.943 INFO:tasks.workunit.client.0.trial112.stdout: "kb_used_omap": 6, 2026-02-13T02:22:47.944 INFO:tasks.workunit.client.0.trial112.stdout: "kb_used_meta": 28089, 2026-02-13T02:22:47.944 INFO:tasks.workunit.client.0.trial112.stdout: "kb_avail": 104829112, 2026-02-13T02:22:47.944 INFO:tasks.workunit.client.0.trial112.stdout: "statfs": { 2026-02-13T02:22:47.944 INFO:tasks.workunit.client.0.trial112.stdout: "total": 107374182400, 2026-02-13T02:22:47.944 INFO:tasks.workunit.client.0.trial112.stdout: "available": 107345010688, 2026-02-13T02:22:47.944 INFO:tasks.workunit.client.0.trial112.stdout: "internally_reserved": 0, 2026-02-13T02:22:47.944 INFO:tasks.workunit.client.0.trial112.stdout: "allocated": 385024, 2026-02-13T02:22:47.944 INFO:tasks.workunit.client.0.trial112.stdout: "data_stored": 206593, 2026-02-13T02:22:47.944 INFO:tasks.workunit.client.0.trial112.stdout: "data_compressed": 0, 2026-02-13T02:22:47.944 INFO:tasks.workunit.client.0.trial112.stdout: "data_compressed_allocated": 0, 2026-02-13T02:22:47.944 INFO:tasks.workunit.client.0.trial112.stdout: "data_compressed_original": 0, 2026-02-13T02:22:47.944 INFO:tasks.workunit.client.0.trial112.stdout: "omap_allocated": 6355, 2026-02-13T02:22:47.945 INFO:tasks.workunit.client.0.trial112.stdout: "internal_metadata": 28763949 2026-02-13T02:22:47.945 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:22:47.945 INFO:tasks.workunit.client.0.trial112.stdout: "hb_peers": [ 2026-02-13T02:22:47.945 INFO:tasks.workunit.client.0.trial112.stdout: 1, 2026-02-13T02:22:47.945 INFO:tasks.workunit.client.0.trial112.stdout: 2 2026-02-13T02:22:47.945 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:22:47.945 INFO:tasks.workunit.client.0.trial112.stdout: "snap_trim_queue_len": 0, 2026-02-13T02:22:47.945 INFO:tasks.workunit.client.0.trial112.stdout: "num_snap_trimming": 0, 2026-02-13T02:22:47.945 INFO:tasks.workunit.client.0.trial112.stdout: "num_shards_repaired": 13, 2026-02-13T02:22:47.945 INFO:tasks.workunit.client.0.trial112.stdout: "op_queue_age_hist": { 2026-02-13T02:22:47.945 INFO:tasks.workunit.client.0.trial112.stdout: "histogram": [], 2026-02-13T02:22:47.945 INFO:tasks.workunit.client.0.trial112.stdout: "upper_bound": 1 2026-02-13T02:22:47.945 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:22:47.946 INFO:tasks.workunit.client.0.trial112.stdout: "perf_stat": { 2026-02-13T02:22:47.946 INFO:tasks.workunit.client.0.trial112.stdout: "commit_latency_ms": 0, 2026-02-13T02:22:47.946 INFO:tasks.workunit.client.0.trial112.stdout: "apply_latency_ms": 0, 2026-02-13T02:22:47.946 INFO:tasks.workunit.client.0.trial112.stdout: "commit_latency_ns": 0, 2026-02-13T02:22:47.946 INFO:tasks.workunit.client.0.trial112.stdout: "apply_latency_ns": 0 2026-02-13T02:22:47.946 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:22:47.946 INFO:tasks.workunit.client.0.trial112.stdout: "alerts": [] 2026-02-13T02:22:47.946 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:22:47.947 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2026-02-13T02:22:47.947 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 0 ).num_shards_repaired' 2026-02-13T02:22:48.091 INFO:tasks.workunit.client.0.trial112.stderr:dumped all 2026-02-13T02:22:48.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_repair_stats_ec: COUNT=13 2026-02-13T02:22:48.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:848: TEST_repair_stats_ec: test 13 = 13 2026-02-13T02:22:48.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:839: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:22:48.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:840: TEST_repair_stats_ec: '[' 1 = 0 -o 1 = 1 ']' 2026-02-13T02:22:48.102 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:841: TEST_repair_stats_ec: expr 26 / 2 2026-02-13T02:22:48.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:841: TEST_repair_stats_ec: repair=13 2026-02-13T02:22:48.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:846: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2026-02-13T02:22:48.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:846: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 1 )' 2026-02-13T02:22:48.250 INFO:tasks.workunit.client.0.trial112.stderr:dumped all 2026-02-13T02:22:48.259 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:22:48.259 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 1, 2026-02-13T02:22:48.259 INFO:tasks.workunit.client.0.trial112.stdout: "up_from": 26, 2026-02-13T02:22:48.259 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 111669149703, 2026-02-13T02:22:48.259 INFO:tasks.workunit.client.0.trial112.stdout: "num_pgs": 1, 2026-02-13T02:22:48.259 INFO:tasks.workunit.client.0.trial112.stdout: "num_osds": 1, 2026-02-13T02:22:48.259 INFO:tasks.workunit.client.0.trial112.stdout: "num_per_pool_osds": 1, 2026-02-13T02:22:48.260 INFO:tasks.workunit.client.0.trial112.stdout: "num_per_pool_omap_osds": 1, 2026-02-13T02:22:48.260 INFO:tasks.workunit.client.0.trial112.stdout: "kb": 104857600, 2026-02-13T02:22:48.260 INFO:tasks.workunit.client.0.trial112.stdout: "kb_used": 28488, 2026-02-13T02:22:48.260 INFO:tasks.workunit.client.0.trial112.stdout: "kb_used_data": 376, 2026-02-13T02:22:48.260 INFO:tasks.workunit.client.0.trial112.stdout: "kb_used_omap": 8, 2026-02-13T02:22:48.260 INFO:tasks.workunit.client.0.trial112.stdout: "kb_used_meta": 28087, 2026-02-13T02:22:48.260 INFO:tasks.workunit.client.0.trial112.stdout: "kb_avail": 104829112, 2026-02-13T02:22:48.260 INFO:tasks.workunit.client.0.trial112.stdout: "statfs": { 2026-02-13T02:22:48.260 INFO:tasks.workunit.client.0.trial112.stdout: "total": 107374182400, 2026-02-13T02:22:48.260 INFO:tasks.workunit.client.0.trial112.stdout: "available": 107345010688, 2026-02-13T02:22:48.260 INFO:tasks.workunit.client.0.trial112.stdout: "internally_reserved": 0, 2026-02-13T02:22:48.260 INFO:tasks.workunit.client.0.trial112.stdout: "allocated": 385024, 2026-02-13T02:22:48.261 INFO:tasks.workunit.client.0.trial112.stdout: "data_stored": 206593, 2026-02-13T02:22:48.261 INFO:tasks.workunit.client.0.trial112.stdout: "data_compressed": 0, 2026-02-13T02:22:48.261 INFO:tasks.workunit.client.0.trial112.stdout: "data_compressed_allocated": 0, 2026-02-13T02:22:48.261 INFO:tasks.workunit.client.0.trial112.stdout: "data_compressed_original": 0, 2026-02-13T02:22:48.261 INFO:tasks.workunit.client.0.trial112.stdout: "omap_allocated": 8310, 2026-02-13T02:22:48.261 INFO:tasks.workunit.client.0.trial112.stdout: "internal_metadata": 28761994 2026-02-13T02:22:48.261 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:22:48.261 INFO:tasks.workunit.client.0.trial112.stdout: "hb_peers": [ 2026-02-13T02:22:48.261 INFO:tasks.workunit.client.0.trial112.stdout: 0, 2026-02-13T02:22:48.261 INFO:tasks.workunit.client.0.trial112.stdout: 2 2026-02-13T02:22:48.261 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:22:48.261 INFO:tasks.workunit.client.0.trial112.stdout: "snap_trim_queue_len": 0, 2026-02-13T02:22:48.261 INFO:tasks.workunit.client.0.trial112.stdout: "num_snap_trimming": 0, 2026-02-13T02:22:48.262 INFO:tasks.workunit.client.0.trial112.stdout: "num_shards_repaired": 13, 2026-02-13T02:22:48.262 INFO:tasks.workunit.client.0.trial112.stdout: "op_queue_age_hist": { 2026-02-13T02:22:48.262 INFO:tasks.workunit.client.0.trial112.stdout: "histogram": [], 2026-02-13T02:22:48.262 INFO:tasks.workunit.client.0.trial112.stdout: "upper_bound": 1 2026-02-13T02:22:48.262 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:22:48.262 INFO:tasks.workunit.client.0.trial112.stdout: "perf_stat": { 2026-02-13T02:22:48.262 INFO:tasks.workunit.client.0.trial112.stdout: "commit_latency_ms": 0, 2026-02-13T02:22:48.262 INFO:tasks.workunit.client.0.trial112.stdout: "apply_latency_ms": 0, 2026-02-13T02:22:48.262 INFO:tasks.workunit.client.0.trial112.stdout: "commit_latency_ns": 0, 2026-02-13T02:22:48.262 INFO:tasks.workunit.client.0.trial112.stdout: "apply_latency_ns": 0 2026-02-13T02:22:48.262 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:22:48.262 INFO:tasks.workunit.client.0.trial112.stdout: "alerts": [] 2026-02-13T02:22:48.263 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:22:48.263 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2026-02-13T02:22:48.263 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 1 ).num_shards_repaired' 2026-02-13T02:22:48.416 INFO:tasks.workunit.client.0.trial112.stderr:dumped all 2026-02-13T02:22:48.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_repair_stats_ec: COUNT=13 2026-02-13T02:22:48.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:848: TEST_repair_stats_ec: test 13 = 13 2026-02-13T02:22:48.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:839: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:22:48.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:840: TEST_repair_stats_ec: '[' 2 = 0 -o 2 = 1 ']' 2026-02-13T02:22:48.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:843: TEST_repair_stats_ec: repair=0 2026-02-13T02:22:48.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:846: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2026-02-13T02:22:48.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:846: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 2 )' 2026-02-13T02:22:48.575 INFO:tasks.workunit.client.0.trial112.stderr:dumped all 2026-02-13T02:22:48.585 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:22:48.585 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 2, 2026-02-13T02:22:48.585 INFO:tasks.workunit.client.0.trial112.stdout: "up_from": 14, 2026-02-13T02:22:48.585 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 60129542155, 2026-02-13T02:22:48.585 INFO:tasks.workunit.client.0.trial112.stdout: "num_pgs": 1, 2026-02-13T02:22:48.585 INFO:tasks.workunit.client.0.trial112.stdout: "num_osds": 1, 2026-02-13T02:22:48.585 INFO:tasks.workunit.client.0.trial112.stdout: "num_per_pool_osds": 1, 2026-02-13T02:22:48.585 INFO:tasks.workunit.client.0.trial112.stdout: "num_per_pool_omap_osds": 1, 2026-02-13T02:22:48.585 INFO:tasks.workunit.client.0.trial112.stdout: "kb": 104857600, 2026-02-13T02:22:48.585 INFO:tasks.workunit.client.0.trial112.stdout: "kb_used": 27336, 2026-02-13T02:22:48.585 INFO:tasks.workunit.client.0.trial112.stdout: "kb_used_data": 376, 2026-02-13T02:22:48.586 INFO:tasks.workunit.client.0.trial112.stdout: "kb_used_omap": 11, 2026-02-13T02:22:48.586 INFO:tasks.workunit.client.0.trial112.stdout: "kb_used_meta": 26932, 2026-02-13T02:22:48.586 INFO:tasks.workunit.client.0.trial112.stdout: "kb_avail": 104830264, 2026-02-13T02:22:48.586 INFO:tasks.workunit.client.0.trial112.stdout: "statfs": { 2026-02-13T02:22:48.586 INFO:tasks.workunit.client.0.trial112.stdout: "total": 107374182400, 2026-02-13T02:22:48.586 INFO:tasks.workunit.client.0.trial112.stdout: "available": 107346190336, 2026-02-13T02:22:48.586 INFO:tasks.workunit.client.0.trial112.stdout: "internally_reserved": 0, 2026-02-13T02:22:48.586 INFO:tasks.workunit.client.0.trial112.stdout: "allocated": 385024, 2026-02-13T02:22:48.586 INFO:tasks.workunit.client.0.trial112.stdout: "data_stored": 206593, 2026-02-13T02:22:48.586 INFO:tasks.workunit.client.0.trial112.stdout: "data_compressed": 0, 2026-02-13T02:22:48.586 INFO:tasks.workunit.client.0.trial112.stdout: "data_compressed_allocated": 0, 2026-02-13T02:22:48.586 INFO:tasks.workunit.client.0.trial112.stdout: "data_compressed_original": 0, 2026-02-13T02:22:48.587 INFO:tasks.workunit.client.0.trial112.stdout: "omap_allocated": 12041, 2026-02-13T02:22:48.587 INFO:tasks.workunit.client.0.trial112.stdout: "internal_metadata": 27578615 2026-02-13T02:22:48.587 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:22:48.587 INFO:tasks.workunit.client.0.trial112.stdout: "hb_peers": [ 2026-02-13T02:22:48.587 INFO:tasks.workunit.client.0.trial112.stdout: 0, 2026-02-13T02:22:48.587 INFO:tasks.workunit.client.0.trial112.stdout: 1 2026-02-13T02:22:48.587 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:22:48.587 INFO:tasks.workunit.client.0.trial112.stdout: "snap_trim_queue_len": 0, 2026-02-13T02:22:48.587 INFO:tasks.workunit.client.0.trial112.stdout: "num_snap_trimming": 0, 2026-02-13T02:22:48.587 INFO:tasks.workunit.client.0.trial112.stdout: "num_shards_repaired": 0, 2026-02-13T02:22:48.587 INFO:tasks.workunit.client.0.trial112.stdout: "op_queue_age_hist": { 2026-02-13T02:22:48.587 INFO:tasks.workunit.client.0.trial112.stdout: "histogram": [], 2026-02-13T02:22:48.588 INFO:tasks.workunit.client.0.trial112.stdout: "upper_bound": 1 2026-02-13T02:22:48.588 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:22:48.588 INFO:tasks.workunit.client.0.trial112.stdout: "perf_stat": { 2026-02-13T02:22:48.588 INFO:tasks.workunit.client.0.trial112.stdout: "commit_latency_ms": 0, 2026-02-13T02:22:48.588 INFO:tasks.workunit.client.0.trial112.stdout: "apply_latency_ms": 0, 2026-02-13T02:22:48.588 INFO:tasks.workunit.client.0.trial112.stdout: "commit_latency_ns": 0, 2026-02-13T02:22:48.588 INFO:tasks.workunit.client.0.trial112.stdout: "apply_latency_ns": 0 2026-02-13T02:22:48.588 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:22:48.588 INFO:tasks.workunit.client.0.trial112.stdout: "alerts": [] 2026-02-13T02:22:48.588 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:22:48.589 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2026-02-13T02:22:48.589 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 2 ).num_shards_repaired' 2026-02-13T02:22:48.739 INFO:tasks.workunit.client.0.trial112.stderr:dumped all 2026-02-13T02:22:48.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_repair_stats_ec: COUNT=0 2026-02-13T02:22:48.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:848: TEST_repair_stats_ec: test 0 = 0 2026-02-13T02:22:48.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:851: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2026-02-13T02:22:48.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:851: TEST_repair_stats_ec: jq .pg_map.osd_stats_sum 2026-02-13T02:22:48.897 INFO:tasks.workunit.client.0.trial112.stderr:dumped all 2026-02-13T02:22:48.906 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:22:48.906 INFO:tasks.workunit.client.0.trial112.stdout: "up_from": 0, 2026-02-13T02:22:48.906 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 0, 2026-02-13T02:22:48.906 INFO:tasks.workunit.client.0.trial112.stdout: "num_pgs": 3, 2026-02-13T02:22:48.906 INFO:tasks.workunit.client.0.trial112.stdout: "num_osds": 3, 2026-02-13T02:22:48.906 INFO:tasks.workunit.client.0.trial112.stdout: "num_per_pool_osds": 3, 2026-02-13T02:22:48.906 INFO:tasks.workunit.client.0.trial112.stdout: "num_per_pool_omap_osds": 3, 2026-02-13T02:22:48.907 INFO:tasks.workunit.client.0.trial112.stdout: "kb": 314572800, 2026-02-13T02:22:48.907 INFO:tasks.workunit.client.0.trial112.stdout: "kb_used": 84312, 2026-02-13T02:22:48.907 INFO:tasks.workunit.client.0.trial112.stdout: "kb_used_data": 1128, 2026-02-13T02:22:48.907 INFO:tasks.workunit.client.0.trial112.stdout: "kb_used_omap": 26, 2026-02-13T02:22:48.907 INFO:tasks.workunit.client.0.trial112.stdout: "kb_used_meta": 83109, 2026-02-13T02:22:48.907 INFO:tasks.workunit.client.0.trial112.stdout: "kb_avail": 314488488, 2026-02-13T02:22:48.907 INFO:tasks.workunit.client.0.trial112.stdout: "statfs": { 2026-02-13T02:22:48.907 INFO:tasks.workunit.client.0.trial112.stdout: "total": 322122547200, 2026-02-13T02:22:48.907 INFO:tasks.workunit.client.0.trial112.stdout: "available": 322036211712, 2026-02-13T02:22:48.907 INFO:tasks.workunit.client.0.trial112.stdout: "internally_reserved": 0, 2026-02-13T02:22:48.907 INFO:tasks.workunit.client.0.trial112.stdout: "allocated": 1155072, 2026-02-13T02:22:48.907 INFO:tasks.workunit.client.0.trial112.stdout: "data_stored": 619779, 2026-02-13T02:22:48.908 INFO:tasks.workunit.client.0.trial112.stdout: "data_compressed": 0, 2026-02-13T02:22:48.908 INFO:tasks.workunit.client.0.trial112.stdout: "data_compressed_allocated": 0, 2026-02-13T02:22:48.908 INFO:tasks.workunit.client.0.trial112.stdout: "data_compressed_original": 0, 2026-02-13T02:22:48.908 INFO:tasks.workunit.client.0.trial112.stdout: "omap_allocated": 26706, 2026-02-13T02:22:48.908 INFO:tasks.workunit.client.0.trial112.stdout: "internal_metadata": 85104558 2026-02-13T02:22:48.908 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:22:48.908 INFO:tasks.workunit.client.0.trial112.stdout: "hb_peers": [], 2026-02-13T02:22:48.908 INFO:tasks.workunit.client.0.trial112.stdout: "snap_trim_queue_len": 0, 2026-02-13T02:22:48.908 INFO:tasks.workunit.client.0.trial112.stdout: "num_snap_trimming": 0, 2026-02-13T02:22:48.908 INFO:tasks.workunit.client.0.trial112.stdout: "num_shards_repaired": 26, 2026-02-13T02:22:48.908 INFO:tasks.workunit.client.0.trial112.stdout: "op_queue_age_hist": { 2026-02-13T02:22:48.908 INFO:tasks.workunit.client.0.trial112.stdout: "histogram": [], 2026-02-13T02:22:48.909 INFO:tasks.workunit.client.0.trial112.stdout: "upper_bound": 1 2026-02-13T02:22:48.909 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:22:48.909 INFO:tasks.workunit.client.0.trial112.stdout: "perf_stat": { 2026-02-13T02:22:48.909 INFO:tasks.workunit.client.0.trial112.stdout: "commit_latency_ms": 0, 2026-02-13T02:22:48.909 INFO:tasks.workunit.client.0.trial112.stdout: "apply_latency_ms": 0, 2026-02-13T02:22:48.909 INFO:tasks.workunit.client.0.trial112.stdout: "commit_latency_ns": 0, 2026-02-13T02:22:48.909 INFO:tasks.workunit.client.0.trial112.stdout: "apply_latency_ns": 0 2026-02-13T02:22:48.909 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:22:48.909 INFO:tasks.workunit.client.0.trial112.stdout: "alerts": [], 2026-02-13T02:22:48.909 INFO:tasks.workunit.client.0.trial112.stdout: "network_ping_times": [] 2026-02-13T02:22:48.909 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:22:48.910 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:852: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2026-02-13T02:22:48.910 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:852: TEST_repair_stats_ec: jq .pg_map.osd_stats_sum.num_shards_repaired 2026-02-13T02:22:49.057 INFO:tasks.workunit.client.0.trial112.stderr:dumped all 2026-02-13T02:22:49.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:852: TEST_repair_stats_ec: COUNT=26 2026-02-13T02:22:49.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:853: TEST_repair_stats_ec: test 26 = 26 2026-02-13T02:22:49.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-13T02:22:49.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:22:49.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:22:49.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:22:49.071 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:22:49.071 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:22:49.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:22:49.072 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:22:49.072 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:22:49.383 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:22:49.384 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:22:49.385 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:22:49.386 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:22:49.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:22:49.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:22:49.389 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:22:49.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:22:49.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:22:49.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:22:49.391 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:22:49.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:22:49.393 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:22:49.393 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:22:49.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:22:49.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:22:49.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:22:49.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:22:49.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:22:49.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:22:49.410 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-13T02:22:49.410 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-13T02:22:49.410 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-13T02:22:49.410 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-13T02:22:49.410 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:22:49.410 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:22:49.410 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:22:49.410 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:22:49.410 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:22:49.410 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:22:49.410 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:22:49.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:22:49.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:22:49.413 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:22:49.414 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:22:49.415 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:22:49.416 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:22:49.416 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:22:49.416 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:22:49.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:22:49.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:22:49.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:22:49.419 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:22:49.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:22:49.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:22:49.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:22:49.423 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:22:49.423 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:22:49.423 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:22:49.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:22:49.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:22:49.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:22:49.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-13T02:22:49.426 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T02:22:49.426 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:22:49.426 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:22:49.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.35207 2026-02-13T02:22:49.428 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T02:22:49.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-13T02:22:49.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T02:22:49.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-13T02:22:49.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_request_scrub_priority td/osd-scrub-repair 2026-02-13T02:22:49.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6369: TEST_request_scrub_priority: local dir=td/osd-scrub-repair 2026-02-13T02:22:49.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6370: TEST_request_scrub_priority: local poolname=psr_pool 2026-02-13T02:22:49.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6371: TEST_request_scrub_priority: local objname=POBJ 2026-02-13T02:22:49.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6372: TEST_request_scrub_priority: local OBJECTS=64 2026-02-13T02:22:49.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6373: TEST_request_scrub_priority: local PGS=8 2026-02-13T02:22:49.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6375: TEST_request_scrub_priority: run_mon td/osd-scrub-repair a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-13T02:22:49.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-13T02:22:49.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-13T02:22:49.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-13T02:22:49.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-13T02:22:49.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-13T02:22:49.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-13T02:22:49.452 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-13T02:22:49.453 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:22:49.453 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:22:49.453 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:22:49.453 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:22:49.453 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:22:49.454 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:22:49.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-13T02:22:49.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-13T02:22:49.482 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-13T02:22:49.482 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:22:49.483 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:22:49.483 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-13T02:22:49.484 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-13T02:22:49.484 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:22:49.484 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:22:49.484 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:22:49.484 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:22:49.484 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:22:49.484 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:22:49.485 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:22:49.485 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:22:49.485 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get fsid 2026-02-13T02:22:49.541 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-13T02:22:49.541 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:22:49.541 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:22:49.542 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-13T02:22:49.542 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-13T02:22:49.542 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:22:49.542 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:22:49.543 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:22:49.543 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:22:49.544 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:22:49.544 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:22:49.544 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:22:49.544 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:22:49.544 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get mon_host 2026-02-13T02:22:49.600 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6376: TEST_request_scrub_priority: run_mgr td/osd-scrub-repair x 2026-02-13T02:22:49.601 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-13T02:22:49.601 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-13T02:22:49.601 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-13T02:22:49.601 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-13T02:22:49.601 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-13T02:22:49.601 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-13T02:22:49.700 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-13T02:22:49.700 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:22:49.700 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:22:49.701 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:22:49.701 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:22:49.701 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:22:49.701 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:22:49.702 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:22:49.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:22:49.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6377: TEST_request_scrub_priority: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 ' 2026-02-13T02:22:49.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6378: TEST_request_scrub_priority: ceph_osd_args+=--osd_scrub_backoff_ratio=0 2026-02-13T02:22:49.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6379: TEST_request_scrub_priority: run_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2026-02-13T02:22:49.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:22:49.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:22:49.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-13T02:22:49.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:22:49.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:22:49.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:22:49.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:22:49.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:22:49.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:22:49.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:22:49.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:22:49.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:22:49.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:22:49.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:22:49.726 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:22:49.726 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:22:49.726 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:22:49.727 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:22:49.727 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:22:49.727 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:22:49.727 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:22:49.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:22:49.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:22:49.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:22:49.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:22:49.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:22:49.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:22:49.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:22:49.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:22:49.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:22:49.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:22:49.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:22:49.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2026-02-13T02:22:49.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:22:49.729 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:22:49.729 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=29cd475a-b86d-47b9-a43b-c1af97de0597 2026-02-13T02:22:49.729 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 29cd475a-b86d-47b9-a43b-c1af97de0597' 2026-02-13T02:22:49.729 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 29cd475a-b86d-47b9-a43b-c1af97de0597 2026-02-13T02:22:49.729 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:22:49.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD5io5pzcCJKxAAB5CnjEIA2GwvWA/3BrbwMw== 2026-02-13T02:22:49.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD5io5pzcCJKxAAB5CnjEIA2GwvWA/3BrbwMw=="}' 2026-02-13T02:22:49.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 29cd475a-b86d-47b9-a43b-c1af97de0597 -i td/osd-scrub-repair/0/new.json 2026-02-13T02:22:49.829 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:22:49.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-13T02:22:49.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --mkfs --key AQD5io5pzcCJKxAAB5CnjEIA2GwvWA/3BrbwMw== --osd-uuid 29cd475a-b86d-47b9-a43b-c1af97de0597 2026-02-13T02:22:49.854 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:49.839+0000 7fe5914e9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:22:49.856 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:49.843+0000 7fe5914e9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:22:49.856 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:49.843+0000 7fe5914e9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:22:49.857 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:49.843+0000 7fe5914e9a40 -1 bdev(0x5639a5cfcc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-13T02:22:49.857 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:49.843+0000 7fe5914e9a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-13T02:22:50.580 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-13T02:22:50.580 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:22:50.581 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-13T02:22:50.581 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T02:22:50.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:22:50.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-13T02:22:50.688 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T02:22:50.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2026-02-13T02:22:50.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:22:50.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:22:50.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:22:50.714 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:50.699+0000 7f08146d7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:22:50.716 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:50.703+0000 7f08146d7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:22:50.718 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:50.703+0000 7f08146d7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:22:50.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-13T02:22:50.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:22:50.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:22:50.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:22:50.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:22:50.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:22:50.783 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:22:50.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:22:50.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:22:50.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:22:50.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:22:51.205 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:51.191+0000 7f08146d7a40 -1 Falling back to public interface 2026-02-13T02:22:51.876 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:22:51.876 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:22:51.876 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:22:51.877 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:22:51.877 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:22:51.877 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:22:51.970 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:22:52.145 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:22:52.131+0000 7f08146d7a40 -1 osd.0 0 log_to_monitors true 2026-02-13T02:22:52.971 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:22:52.971 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:22:52.972 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:22:52.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:22:52.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:22:52.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:22:53.137 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3602308971,v1:127.0.0.1:6801/3602308971] [v2:127.0.0.1:6802/3602308971,v1:127.0.0.1:6803/3602308971] exists,up 29cd475a-b86d-47b9-a43b-c1af97de0597 2026-02-13T02:22:53.138 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:22:53.138 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:22:53.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:22:53.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6381: TEST_request_scrub_priority: create_pool psr_pool 8 8 2026-02-13T02:22:53.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool 8 8 2026-02-13T02:22:53.317 INFO:tasks.workunit.client.0.trial112.stderr:pool 'psr_pool' created 2026-02-13T02:22:53.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:22:54.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6382: TEST_request_scrub_priority: wait_for_clean 2026-02-13T02:22:54.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:22:54.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:22:54.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:22:54.329 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:22:54.330 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:22:54.330 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:22:54.331 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:22:54.331 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:22:54.331 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:22:54.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:22:54.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:22:54.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:22:54.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:22:54.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:22:54.452 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:22:54.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-13T02:22:54.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:22:54.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:22:54.614 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:22:54.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836482 2026-02-13T02:22:54.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836482 2026-02-13T02:22:54.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836482' 2026-02-13T02:22:54.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:22:54.689 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836482 2026-02-13T02:22:54.689 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:22:54.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:22:54.692 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836482 2026-02-13T02:22:54.692 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:22:54.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836482 2026-02-13T02:22:54.694 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836482' 2026-02-13T02:22:54.694 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836482 2026-02-13T02:22:54.694 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:22:54.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836481 -lt 21474836482 2026-02-13T02:22:54.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:22:55.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:22:55.865 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:22:56.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836482 2026-02-13T02:22:56.026 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:22:56.026 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:22:56.027 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:22:56.218 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 8 == 0 2026-02-13T02:22:56.218 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:22:56.219 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:22:56.219 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:22:56.219 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:22:56.219 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:22:56.219 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:22:56.220 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:22:56.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=8 2026-02-13T02:22:56.380 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:22:56.381 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:22:56.381 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:22:56.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 8 = 8 2026-02-13T02:22:56.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:22:56.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:22:56.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6384: TEST_request_scrub_priority: local osd=0 2026-02-13T02:22:56.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6385: TEST_request_scrub_priority: add_something td/osd-scrub-repair psr_pool POBJ noscrub 2026-02-13T02:22:56.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:22:56.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=psr_pool 2026-02-13T02:22:56.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=POBJ 2026-02-13T02:22:56.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:22:56.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:22:56.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:22:56.826 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:22:56.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:22:57.027 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:22:57.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:22:57.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:22:57.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool psr_pool put POBJ td/osd-scrub-repair/ORIGINAL 2026-02-13T02:22:57.066 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6386: TEST_request_scrub_priority: get_primary psr_pool POBJ 2026-02-13T02:22:57.066 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=psr_pool 2026-02-13T02:22:57.066 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=POBJ 2026-02-13T02:22:57.066 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map psr_pool POBJ 2026-02-13T02:22:57.066 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T02:22:57.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6386: TEST_request_scrub_priority: local primary=0 2026-02-13T02:22:57.232 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6387: TEST_request_scrub_priority: get_pg psr_pool POBJ 2026-02-13T02:22:57.232 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=psr_pool 2026-02-13T02:22:57.232 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=POBJ 2026-02-13T02:22:57.232 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map psr_pool POBJ 2026-02-13T02:22:57.233 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T02:22:57.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6387: TEST_request_scrub_priority: local pg=1.1 2026-02-13T02:22:57.399 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6388: TEST_request_scrub_priority: ceph osd dump 2026-02-13T02:22:57.399 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6388: TEST_request_scrub_priority: grep '^pool.*['\'']psr_pool['\'']' 2026-02-13T02:22:57.399 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6388: TEST_request_scrub_priority: awk '{ print $2 }' 2026-02-13T02:22:57.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6388: TEST_request_scrub_priority: poolid=1 2026-02-13T02:22:57.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6390: TEST_request_scrub_priority: local otherpgs 2026-02-13T02:22:57.570 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6391: TEST_request_scrub_priority: expr 8 - 1 2026-02-13T02:22:57.571 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6391: TEST_request_scrub_priority: seq 0 7 2026-02-13T02:22:57.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6391: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2026-02-13T02:22:57.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6393: TEST_request_scrub_priority: opg=1.0 2026-02-13T02:22:57.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6394: TEST_request_scrub_priority: '[' 1.0 = 1.1 ']' 2026-02-13T02:22:57.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6397: TEST_request_scrub_priority: otherpgs='1.0 ' 2026-02-13T02:22:57.573 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2026-02-13T02:22:57.573 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2026-02-13T02:22:57.573 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:22:57.574 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:22:57.574 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2026-02-13T02:22:57.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: local other_last_scrub=2026-02-13T02:22:53.306110+0000 2026-02-13T02:22:57.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6400: TEST_request_scrub_priority: ceph tell 1.0 schedule-scrub 1.0 2026-02-13T02:22:57.801 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:22:57.801 INFO:tasks.workunit.client.0.trial112.stdout: "deep": false, 2026-02-13T02:22:57.801 INFO:tasks.workunit.client.0.trial112.stdout: "must": false, 2026-02-13T02:22:57.801 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "2026-02-11T02:22:57.793502+0000" 2026-02-13T02:22:57.801 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:22:57.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6391: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2026-02-13T02:22:57.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6393: TEST_request_scrub_priority: opg=1.1 2026-02-13T02:22:57.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6394: TEST_request_scrub_priority: '[' 1.1 = 1.1 ']' 2026-02-13T02:22:57.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6395: TEST_request_scrub_priority: continue 2026-02-13T02:22:57.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6391: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2026-02-13T02:22:57.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6393: TEST_request_scrub_priority: opg=1.2 2026-02-13T02:22:57.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6394: TEST_request_scrub_priority: '[' 1.2 = 1.1 ']' 2026-02-13T02:22:57.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6397: TEST_request_scrub_priority: otherpgs='1.0 1.2 ' 2026-02-13T02:22:57.809 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2026-02-13T02:22:57.809 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2026-02-13T02:22:57.809 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:22:57.810 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:22:57.810 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2026-02-13T02:22:57.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: local other_last_scrub=2026-02-13T02:22:53.306110+0000 2026-02-13T02:22:57.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6400: TEST_request_scrub_priority: ceph tell 1.2 schedule-scrub 1.2 2026-02-13T02:22:58.039 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:22:58.039 INFO:tasks.workunit.client.0.trial112.stdout: "deep": false, 2026-02-13T02:22:58.039 INFO:tasks.workunit.client.0.trial112.stdout: "must": false, 2026-02-13T02:22:58.039 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "2026-02-11T02:22:58.031466+0000" 2026-02-13T02:22:58.039 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:22:58.047 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6391: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2026-02-13T02:22:58.047 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6393: TEST_request_scrub_priority: opg=1.3 2026-02-13T02:22:58.047 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6394: TEST_request_scrub_priority: '[' 1.3 = 1.1 ']' 2026-02-13T02:22:58.047 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6397: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 ' 2026-02-13T02:22:58.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2026-02-13T02:22:58.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2026-02-13T02:22:58.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:22:58.048 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:22:58.048 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2026-02-13T02:22:58.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: local other_last_scrub=2026-02-13T02:22:53.306110+0000 2026-02-13T02:22:58.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6400: TEST_request_scrub_priority: ceph tell 1.3 schedule-scrub 1.3 2026-02-13T02:22:58.277 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:22:58.277 INFO:tasks.workunit.client.0.trial112.stdout: "deep": false, 2026-02-13T02:22:58.277 INFO:tasks.workunit.client.0.trial112.stdout: "must": false, 2026-02-13T02:22:58.277 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "2026-02-11T02:22:58.269577+0000" 2026-02-13T02:22:58.278 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:22:58.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6391: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2026-02-13T02:22:58.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6393: TEST_request_scrub_priority: opg=1.4 2026-02-13T02:22:58.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6394: TEST_request_scrub_priority: '[' 1.4 = 1.1 ']' 2026-02-13T02:22:58.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6397: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 ' 2026-02-13T02:22:58.285 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2026-02-13T02:22:58.285 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2026-02-13T02:22:58.285 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:22:58.286 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:22:58.286 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2026-02-13T02:22:58.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: local other_last_scrub=2026-02-13T02:22:53.306110+0000 2026-02-13T02:22:58.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6400: TEST_request_scrub_priority: ceph tell 1.4 schedule-scrub 1.4 2026-02-13T02:22:58.516 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:22:58.516 INFO:tasks.workunit.client.0.trial112.stdout: "deep": false, 2026-02-13T02:22:58.516 INFO:tasks.workunit.client.0.trial112.stdout: "must": false, 2026-02-13T02:22:58.516 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "2026-02-11T02:22:58.508521+0000" 2026-02-13T02:22:58.516 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:22:58.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6391: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2026-02-13T02:22:58.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6393: TEST_request_scrub_priority: opg=1.5 2026-02-13T02:22:58.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6394: TEST_request_scrub_priority: '[' 1.5 = 1.1 ']' 2026-02-13T02:22:58.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6397: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 1.5 ' 2026-02-13T02:22:58.525 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2026-02-13T02:22:58.525 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2026-02-13T02:22:58.525 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:22:58.525 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:22:58.525 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2026-02-13T02:22:58.681 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: local other_last_scrub=2026-02-13T02:22:53.306110+0000 2026-02-13T02:22:58.681 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6400: TEST_request_scrub_priority: ceph tell 1.5 schedule-scrub 1.5 2026-02-13T02:22:58.748 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:22:58.749 INFO:tasks.workunit.client.0.trial112.stdout: "deep": false, 2026-02-13T02:22:58.749 INFO:tasks.workunit.client.0.trial112.stdout: "must": false, 2026-02-13T02:22:58.749 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "2026-02-11T02:22:58.740657+0000" 2026-02-13T02:22:58.749 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:22:58.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6391: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2026-02-13T02:22:58.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6393: TEST_request_scrub_priority: opg=1.6 2026-02-13T02:22:58.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6394: TEST_request_scrub_priority: '[' 1.6 = 1.1 ']' 2026-02-13T02:22:58.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6397: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 1.5 1.6 ' 2026-02-13T02:22:58.757 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2026-02-13T02:22:58.757 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2026-02-13T02:22:58.757 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:22:58.757 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:22:58.757 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2026-02-13T02:22:58.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: local other_last_scrub=2026-02-13T02:22:53.306110+0000 2026-02-13T02:22:58.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6400: TEST_request_scrub_priority: ceph tell 1.6 schedule-scrub 1.6 2026-02-13T02:22:58.987 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:22:58.988 INFO:tasks.workunit.client.0.trial112.stdout: "deep": false, 2026-02-13T02:22:58.988 INFO:tasks.workunit.client.0.trial112.stdout: "must": false, 2026-02-13T02:22:58.988 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "2026-02-11T02:22:58.979734+0000" 2026-02-13T02:22:58.988 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:22:58.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6391: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2026-02-13T02:22:58.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6393: TEST_request_scrub_priority: opg=1.7 2026-02-13T02:22:58.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6394: TEST_request_scrub_priority: '[' 1.7 = 1.1 ']' 2026-02-13T02:22:58.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6397: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 1.5 1.6 1.7 ' 2026-02-13T02:22:58.996 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2026-02-13T02:22:58.996 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2026-02-13T02:22:58.996 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:22:58.996 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:22:58.996 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2026-02-13T02:22:59.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: local other_last_scrub=2026-02-13T02:22:53.306110+0000 2026-02-13T02:22:59.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6400: TEST_request_scrub_priority: ceph tell 1.7 schedule-scrub 1.7 2026-02-13T02:22:59.224 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:22:59.224 INFO:tasks.workunit.client.0.trial112.stdout: "deep": false, 2026-02-13T02:22:59.224 INFO:tasks.workunit.client.0.trial112.stdout: "must": false, 2026-02-13T02:22:59.224 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "2026-02-11T02:22:59.215888+0000" 2026-02-13T02:22:59.224 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:22:59.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6403: TEST_request_scrub_priority: sleep 15 2026-02-13T02:23:14.233 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6404: TEST_request_scrub_priority: flush_pg_stats 2026-02-13T02:23:14.233 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:23:14.234 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:23:14.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-13T02:23:14.393 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:23:14.393 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:23:14.393 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:23:14.469 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836487 2026-02-13T02:23:14.469 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836487 2026-02-13T02:23:14.469 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487' 2026-02-13T02:23:14.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:23:14.470 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836487 2026-02-13T02:23:14.470 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:23:14.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:23:14.472 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836487 2026-02-13T02:23:14.473 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:23:14.474 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836487 2026-02-13T02:23:14.474 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836487 2026-02-13T02:23:14.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2026-02-13T02:23:14.475 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:23:14.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836487 2026-02-13T02:23:14.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:23:15.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:23:15.651 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:23:15.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836487 2026-02-13T02:23:15.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:23:16.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:23:16.815 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:23:16.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836487 2026-02-13T02:23:16.978 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6407: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2026-02-13T02:23:16.978 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2026-02-13T02:23:16.978 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:23:16.978 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:23:16.978 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2026-02-13T02:23:17.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6407: TEST_request_scrub_priority: local last_scrub=2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:17.136 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6408: TEST_request_scrub_priority: ceph tell 1.1 scrub 2026-02-13T02:23:17.201 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:23:17.201 INFO:tasks.workunit.client.0.trial112.stdout: "deep": false, 2026-02-13T02:23:17.201 INFO:tasks.workunit.client.0.trial112.stdout: "must": true, 2026-02-13T02:23:17.201 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "0.000000" 2026-02-13T02:23:17.201 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:23:17.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6410: TEST_request_scrub_priority: ceph osd unset noscrub 2026-02-13T02:23:17.423 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is unset 2026-02-13T02:23:17.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6411: TEST_request_scrub_priority: ceph osd unset nodeep-scrub 2026-02-13T02:23:17.628 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is unset 2026-02-13T02:23:17.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6413: TEST_request_scrub_priority: wait_for_scrub 1.1 2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:17.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.1 2026-02-13T02:23:17.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:17.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:23:17.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:23:17.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:23:17.640 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2026-02-13T02:23:17.640 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2026-02-13T02:23:17.640 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:23:17.641 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:23:17.641 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2026-02-13T02:23:17.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:22:53.306110+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:17.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:23:18.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:23:18.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:23:18.804 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2026-02-13T02:23:18.804 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2026-02-13T02:23:18.804 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:23:18.805 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2026-02-13T02:23:18.805 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:23:18.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:22:53.306110+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:18.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:23:19.970 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:23:19.971 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:23:19.971 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2026-02-13T02:23:19.972 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2026-02-13T02:23:19.972 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:23:19.972 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2026-02-13T02:23:19.972 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:23:20.138 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:23:17.338840+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:20.138 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:23:20.138 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6415: TEST_request_scrub_priority: for opg in $otherpgs $pg 2026-02-13T02:23:20.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6417: TEST_request_scrub_priority: wait_for_scrub 1.0 2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:20.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-13T02:23:20.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:20.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:23:20.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:23:20.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:23:20.139 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:23:20.139 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:23:20.139 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:23:20.139 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:23:20.139 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:23:20.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:57.793502+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:20.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:23:21.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:23:21.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:23:21.301 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:23:21.302 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:23:21.302 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:23:21.302 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:23:21.302 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:23:21.463 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:57.793502+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:21.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:23:22.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:23:22.465 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:23:22.465 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:23:22.466 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:23:22.466 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:23:22.466 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:23:22.466 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:23:22.627 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:57.793502+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:22.627 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:23:23.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:23:23.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:23:23.629 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:23:23.629 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:23:23.629 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:23:23.629 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:23:23.629 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:23:23.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:57.793502+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:23.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:23:24.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:23:24.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:23:24.791 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:23:24.791 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:23:24.791 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:23:24.792 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:23:24.792 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:23:24.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:57.793502+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:24.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:23:25.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:23:25.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:23:25.955 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:23:25.955 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:23:25.955 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:23:25.955 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:23:25.956 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:23:26.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:57.793502+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:26.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:23:27.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:23:27.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:23:27.114 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:23:27.115 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:23:27.115 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:23:27.115 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:23:27.115 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:23:27.270 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:57.793502+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:27.270 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:23:28.271 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:23:28.271 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:23:28.272 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:23:28.272 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:23:28.272 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:23:28.273 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:23:28.273 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:23:28.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:57.793502+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:28.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:23:29.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:23:29.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:23:29.432 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:23:29.432 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:23:29.432 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:23:29.432 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:23:29.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:23:29.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:57.793502+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:29.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:23:30.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:23:30.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:23:30.593 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:23:30.593 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:23:30.594 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:23:30.594 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:23:30.594 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:23:30.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:57.793502+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:30.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:23:31.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:23:31.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:23:31.758 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:23:31.758 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:23:31.759 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:23:31.759 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:23:31.759 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:23:31.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:57.793502+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:31.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:23:32.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:23:32.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:23:32.923 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:23:32.923 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:23:32.923 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:23:32.923 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:23:32.924 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:23:33.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:57.793502+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:33.086 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:23:34.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:23:34.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:23:34.087 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:23:34.088 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:23:34.088 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:23:34.088 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:23:34.088 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:23:34.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:57.793502+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:34.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:23:35.253 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:23:35.253 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:23:35.254 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:23:35.254 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:23:35.254 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:23:35.254 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:23:35.255 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:23:35.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:57.793502+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:35.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:23:36.414 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:23:36.414 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:23:36.415 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:23:36.415 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:23:36.415 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:23:36.415 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:23:36.416 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:23:36.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:57.793502+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:36.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:23:37.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:23:37.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:23:37.579 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:23:37.579 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:23:37.579 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:23:37.580 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:23:37.580 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:23:37.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:57.793502+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:37.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:23:38.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:23:38.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:23:38.746 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:23:38.746 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:23:38.746 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:23:38.746 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:23:38.747 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:23:38.904 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:57.793502+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:38.904 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:23:39.905 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:23:39.905 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:23:39.906 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:23:39.906 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:23:39.906 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:23:39.907 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:23:39.907 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:23:40.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:57.793502+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:40.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:23:41.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:23:41.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:23:41.063 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:23:41.063 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:23:41.063 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:23:41.063 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:23:41.063 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:23:41.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:57.793502+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:41.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:23:42.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:23:42.222 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:23:42.222 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:23:42.223 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:23:42.223 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:23:42.223 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:23:42.223 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:23:42.385 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:57.793502+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:42.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:23:43.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:23:43.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:23:43.388 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:23:43.388 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:23:43.388 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:23:43.388 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:23:43.388 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:23:43.547 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:57.793502+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:43.548 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:23:44.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:23:44.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:23:44.550 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:23:44.550 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:23:44.550 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:23:44.551 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:23:44.551 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:23:44.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:57.793502+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:44.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:23:45.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:23:45.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:23:45.711 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:23:45.711 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:23:45.711 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:23:45.712 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:23:45.712 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:23:45.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:57.793502+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:45.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:23:46.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:23:46.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:23:46.876 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:23:46.876 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:23:46.876 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:23:46.876 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:23:46.876 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:23:47.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:57.793502+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:47.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:23:48.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:23:48.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:23:48.041 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:23:48.041 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:23:48.041 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:23:48.042 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:23:48.042 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:23:48.210 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:57.793502+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:48.210 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:23:49.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:23:49.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:23:49.212 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:23:49.212 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:23:49.212 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:23:49.213 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:23:49.213 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:23:49.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:57.793502+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:49.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:23:50.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:23:50.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:23:50.376 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:23:50.376 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:23:50.376 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:23:50.377 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:23:50.377 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:23:50.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:57.793502+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:50.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:23:51.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:23:51.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:23:51.542 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:23:51.542 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:23:51.542 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:23:51.543 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:23:51.543 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:23:51.705 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:57.793502+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:51.705 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:23:52.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:23:52.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:23:52.707 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:23:52.707 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:23:52.708 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:23:52.708 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:23:52.708 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:23:52.866 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:57.793502+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:52.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:23:53.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:23:53.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:23:53.870 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:23:53.870 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:23:53.870 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:23:53.870 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:23:53.870 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:23:54.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:57.793502+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:54.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:23:55.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:23:55.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:23:55.031 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:23:55.031 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:23:55.032 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:23:55.032 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:23:55.032 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:23:55.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:57.793502+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:55.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:23:56.198 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:23:56.198 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:23:56.199 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:23:56.199 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:23:56.199 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:23:56.199 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:23:56.199 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:23:56.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:57.793502+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:56.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:23:57.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:23:57.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:23:57.364 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:23:57.364 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:23:57.364 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:23:57.365 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:23:57.365 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:23:57.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:57.793502+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:57.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:23:58.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:23:58.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:23:58.531 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:23:58.532 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:23:58.532 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:23:58.532 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:23:58.532 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:23:58.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:57.793502+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:58.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:23:59.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:23:59.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:23:59.697 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:23:59.697 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:23:59.697 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:23:59.698 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:23:59.698 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:23:59.857 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:57.793502+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:23:59.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:24:00.859 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:24:00.859 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:24:00.859 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:24:00.860 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:24:00.860 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:24:00.860 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:24:00.860 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:24:01.021 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:57.793502+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:24:01.021 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:24:02.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:24:02.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:24:02.023 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:24:02.023 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:24:02.023 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:24:02.024 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:24:02.024 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:24:02.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:57.793502+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:24:02.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:24:03.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:24:03.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:24:03.189 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:24:03.189 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:24:03.189 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:24:03.190 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:24:03.190 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:24:03.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:57.793502+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:24:03.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:24:04.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:24:04.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:24:04.353 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:24:04.353 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:24:04.353 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:24:04.354 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:24:04.354 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:24:04.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:23:59.252713+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:24:04.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:24:04.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6415: TEST_request_scrub_priority: for opg in $otherpgs $pg 2026-02-13T02:24:04.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6417: TEST_request_scrub_priority: wait_for_scrub 1.2 2026-02-13T02:22:53.306110+0000 2026-02-13T02:24:04.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.2 2026-02-13T02:24:04.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:22:53.306110+0000 2026-02-13T02:24:04.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:24:04.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:24:04.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:24:04.518 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.2 last_scrub_stamp 2026-02-13T02:24:04.518 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.2 2026-02-13T02:24:04.518 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:24:04.519 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:24:04.519 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2026-02-13T02:24:04.678 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:24:00.255099+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:24:04.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:24:04.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6415: TEST_request_scrub_priority: for opg in $otherpgs $pg 2026-02-13T02:24:04.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6417: TEST_request_scrub_priority: wait_for_scrub 1.3 2026-02-13T02:22:53.306110+0000 2026-02-13T02:24:04.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.3 2026-02-13T02:24:04.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:22:53.306110+0000 2026-02-13T02:24:04.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:24:04.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:24:04.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:24:04.679 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-13T02:24:04.679 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-13T02:24:04.680 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:24:04.680 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:24:04.680 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-13T02:24:04.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:24:01.218988+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:24:04.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:24:04.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6415: TEST_request_scrub_priority: for opg in $otherpgs $pg 2026-02-13T02:24:04.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6417: TEST_request_scrub_priority: wait_for_scrub 1.4 2026-02-13T02:22:53.306110+0000 2026-02-13T02:24:04.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.4 2026-02-13T02:24:04.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:22:53.306110+0000 2026-02-13T02:24:04.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:24:04.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:24:04.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:24:04.844 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.4 last_scrub_stamp 2026-02-13T02:24:04.844 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.4 2026-02-13T02:24:04.844 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:24:04.844 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:24:04.844 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2026-02-13T02:24:05.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:24:02.264396+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:24:05.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:24:05.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6415: TEST_request_scrub_priority: for opg in $otherpgs $pg 2026-02-13T02:24:05.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6417: TEST_request_scrub_priority: wait_for_scrub 1.5 2026-02-13T02:22:53.306110+0000 2026-02-13T02:24:05.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.5 2026-02-13T02:24:05.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:22:53.306110+0000 2026-02-13T02:24:05.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:24:05.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:24:05.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:24:05.007 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.5 last_scrub_stamp 2026-02-13T02:24:05.008 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.5 2026-02-13T02:24:05.008 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:24:05.008 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:24:05.008 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2026-02-13T02:24:05.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:24:03.270641+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:24:05.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:24:05.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6415: TEST_request_scrub_priority: for opg in $otherpgs $pg 2026-02-13T02:24:05.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6417: TEST_request_scrub_priority: wait_for_scrub 1.6 2026-02-13T02:22:53.306110+0000 2026-02-13T02:24:05.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.6 2026-02-13T02:24:05.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:22:53.306110+0000 2026-02-13T02:24:05.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:24:05.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:24:05.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:24:05.166 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.6 last_scrub_stamp 2026-02-13T02:24:05.166 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.6 2026-02-13T02:24:05.166 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:24:05.166 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:24:05.166 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2026-02-13T02:24:05.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:58.979734+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:24:05.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:24:06.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:24:06.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:24:06.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.6 last_scrub_stamp 2026-02-13T02:24:06.324 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.6 2026-02-13T02:24:06.324 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:24:06.324 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:24:06.324 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2026-02-13T02:24:06.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:58.979734+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:24:06.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:24:07.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:24:07.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:24:07.482 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.6 last_scrub_stamp 2026-02-13T02:24:07.482 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.6 2026-02-13T02:24:07.482 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:24:07.482 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:24:07.483 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2026-02-13T02:24:07.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:58.979734+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:24:07.642 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:24:08.642 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:24:08.642 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:24:08.643 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.6 last_scrub_stamp 2026-02-13T02:24:08.643 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.6 2026-02-13T02:24:08.643 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:24:08.644 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:24:08.644 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2026-02-13T02:24:08.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:22:58.979734+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:24:08.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:24:09.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:24:09.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:24:09.805 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.6 last_scrub_stamp 2026-02-13T02:24:09.805 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.6 2026-02-13T02:24:09.805 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:24:09.805 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:24:09.806 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2026-02-13T02:24:09.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:24:04.231896+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:24:09.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:24:09.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6415: TEST_request_scrub_priority: for opg in $otherpgs $pg 2026-02-13T02:24:09.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6417: TEST_request_scrub_priority: wait_for_scrub 1.7 2026-02-13T02:22:53.306110+0000 2026-02-13T02:24:09.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.7 2026-02-13T02:24:09.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:22:53.306110+0000 2026-02-13T02:24:09.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:24:09.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:24:09.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:24:09.966 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.7 last_scrub_stamp 2026-02-13T02:24:09.966 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.7 2026-02-13T02:24:09.966 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:24:09.967 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:24:09.967 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2026-02-13T02:24:10.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:24:05.244980+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:24:10.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:24:10.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6415: TEST_request_scrub_priority: for opg in $otherpgs $pg 2026-02-13T02:24:10.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6417: TEST_request_scrub_priority: wait_for_scrub 1.1 2026-02-13T02:22:53.306110+0000 2026-02-13T02:24:10.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.1 2026-02-13T02:24:10.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:22:53.306110+0000 2026-02-13T02:24:10.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:24:10.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:24:10.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:24:10.127 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2026-02-13T02:24:10.127 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2026-02-13T02:24:10.127 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:24:10.127 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:24:10.127 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2026-02-13T02:24:10.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:23:17.338840+0000 '>' 2026-02-13T02:22:53.306110+0000 2026-02-13T02:24:10.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:24:10.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6421: TEST_request_scrub_priority: grep 'log_channel.*scrub ok' td/osd-scrub-repair/osd.0.log 2026-02-13T02:24:10.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6421: TEST_request_scrub_priority: grep -v purged_snaps 2026-02-13T02:24:10.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6421: TEST_request_scrub_priority: head -1 2026-02-13T02:24:10.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6421: TEST_request_scrub_priority: sed 's/.*[[]DBG[]]//' 2026-02-13T02:24:10.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6421: TEST_request_scrub_priority: grep -q 1.1 2026-02-13T02:24:10.289 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-13T02:24:10.289 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:24:10.289 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:24:10.289 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:24:10.290 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:24:10.290 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:24:10.290 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:24:10.290 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:24:10.290 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:24:10.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:24:10.401 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:24:10.402 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:24:10.402 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:24:10.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:24:10.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:24:10.405 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:24:10.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:24:10.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:24:10.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:24:10.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:24:10.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:24:10.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:24:10.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:24:10.416 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:24:10.416 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:24:10.416 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:24:10.416 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:24:10.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:24:10.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:24:10.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-13T02:24:10.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-13T02:24:10.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-13T02:24:10.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-13T02:24:10.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:24:10.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:24:10.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:24:10.420 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:24:10.420 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:24:10.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:24:10.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:24:10.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:24:10.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:24:10.423 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:24:10.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:24:10.424 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:24:10.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:24:10.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:24:10.426 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:24:10.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:24:10.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:24:10.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:24:10.428 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:24:10.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:24:10.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:24:10.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:24:10.432 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:24:10.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:24:10.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:24:10.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:24:10.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:24:10.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:24:10.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-13T02:24:10.436 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T02:24:10.436 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:24:10.436 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:24:10.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.35207 2026-02-13T02:24:10.438 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T02:24:10.439 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-13T02:24:10.439 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T02:24:10.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-13T02:24:10.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_scrub_warning td/osd-scrub-repair 2026-02-13T02:24:10.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6011: TEST_scrub_warning: local dir=td/osd-scrub-repair 2026-02-13T02:24:10.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6012: TEST_scrub_warning: local poolname=psr_pool 2026-02-13T02:24:10.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6013: TEST_scrub_warning: local objname=POBJ 2026-02-13T02:24:10.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6014: TEST_scrub_warning: local scrubs=5 2026-02-13T02:24:10.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6015: TEST_scrub_warning: local deep_scrubs=5 2026-02-13T02:24:10.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6016: TEST_scrub_warning: local i1_day=86400 2026-02-13T02:24:10.440 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6017: TEST_scrub_warning: calc 86400 '*' 7 2026-02-13T02:24:10.440 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: calc: awk 'BEGIN{print 86400 * 7}' 2026-02-13T02:24:10.441 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6017: TEST_scrub_warning: local i7_days=604800 2026-02-13T02:24:10.442 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6018: TEST_scrub_warning: calc 86400 '*' 14 2026-02-13T02:24:10.442 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: calc: awk 'BEGIN{print 86400 * 14}' 2026-02-13T02:24:10.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6018: TEST_scrub_warning: local i14_days=1209600 2026-02-13T02:24:10.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6019: TEST_scrub_warning: local overdue=0.5 2026-02-13T02:24:10.445 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6020: TEST_scrub_warning: calc 604800 + 86400 + '(' 604800 '*' 0.5 ')' 2026-02-13T02:24:10.445 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: calc: awk 'BEGIN{print 604800 + 86400 + ( 604800 * 0.5 )}' 2026-02-13T02:24:10.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6020: TEST_scrub_warning: local conf_overdue_seconds=993600 2026-02-13T02:24:10.447 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6021: TEST_scrub_warning: calc 1209600 + 86400 + '(' 1209600 '*' 0.5 ')' 2026-02-13T02:24:10.447 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: calc: awk 'BEGIN{print 1209600 + 86400 + ( 1209600 * 0.5 )}' 2026-02-13T02:24:10.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6021: TEST_scrub_warning: local pool_overdue_seconds=1900800 2026-02-13T02:24:10.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6023: TEST_scrub_warning: run_mon td/osd-scrub-repair a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-13T02:24:10.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-13T02:24:10.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-13T02:24:10.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-13T02:24:10.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-13T02:24:10.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-13T02:24:10.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-13T02:24:10.473 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-13T02:24:10.473 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:24:10.473 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:24:10.474 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:24:10.474 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:24:10.474 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:24:10.474 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:24:10.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-13T02:24:10.505 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-13T02:24:10.506 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-13T02:24:10.506 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:24:10.506 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:24:10.506 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-13T02:24:10.507 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-13T02:24:10.507 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:24:10.507 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:24:10.508 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:24:10.508 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:24:10.508 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:24:10.508 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:24:10.508 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:24:10.508 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:24:10.509 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get fsid 2026-02-13T02:24:10.565 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-13T02:24:10.565 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:24:10.565 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:24:10.565 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-13T02:24:10.566 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-13T02:24:10.566 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:24:10.566 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:24:10.566 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:24:10.567 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:24:10.567 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:24:10.568 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:24:10.568 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:24:10.568 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:24:10.568 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get mon_host 2026-02-13T02:24:10.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6024: TEST_scrub_warning: run_mgr td/osd-scrub-repair x --mon_warn_pg_not_scrubbed_ratio=0.5 --mon_warn_pg_not_deep_scrubbed_ratio=0.5 2026-02-13T02:24:10.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-13T02:24:10.624 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-13T02:24:10.624 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-13T02:24:10.624 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-13T02:24:10.624 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-13T02:24:10.624 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-13T02:24:10.722 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-13T02:24:10.723 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:24:10.723 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:24:10.723 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:24:10.724 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:24:10.724 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:24:10.724 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:24:10.724 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:24:10.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mon_warn_pg_not_scrubbed_ratio=0.5 --mon_warn_pg_not_deep_scrubbed_ratio=0.5 2026-02-13T02:24:10.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6025: TEST_scrub_warning: run_osd td/osd-scrub-repair 0 --osd_scrub_backoff_ratio=0 2026-02-13T02:24:10.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:24:10.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:24:10.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-13T02:24:10.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:24:10.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:24:10.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:24:10.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:24:10.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:24:10.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:24:10.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:24:10.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:24:10.747 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:24:10.747 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:24:10.747 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:24:10.747 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:24:10.747 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:24:10.747 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:24:10.747 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:24:10.747 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:24:10.747 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:24:10.747 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:24:10.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:24:10.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:24:10.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:24:10.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:24:10.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:24:10.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:24:10.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:24:10.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:24:10.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:24:10.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:24:10.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:24:10.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_scrub_backoff_ratio=0 2026-02-13T02:24:10.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:24:10.750 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:24:10.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=47495e47-b606-4abf-bc83-fbee61dba676 2026-02-13T02:24:10.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 47495e47-b606-4abf-bc83-fbee61dba676' 2026-02-13T02:24:10.751 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 47495e47-b606-4abf-bc83-fbee61dba676 2026-02-13T02:24:10.751 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:24:10.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBKi45prRfnLBAA2r71fkE7fpp7YaYbCx9Txg== 2026-02-13T02:24:10.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBKi45prRfnLBAA2r71fkE7fpp7YaYbCx9Txg=="}' 2026-02-13T02:24:10.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 47495e47-b606-4abf-bc83-fbee61dba676 -i td/osd-scrub-repair/0/new.json 2026-02-13T02:24:10.850 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:24:10.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-13T02:24:10.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_backoff_ratio=0 --mkfs --key AQBKi45prRfnLBAA2r71fkE7fpp7YaYbCx9Txg== --osd-uuid 47495e47-b606-4abf-bc83-fbee61dba676 2026-02-13T02:24:10.880 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:24:10.867+0000 7f2b3c5baa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:24:10.881 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:24:10.871+0000 7f2b3c5baa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:24:10.882 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:24:10.871+0000 7f2b3c5baa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:24:10.883 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:24:10.871+0000 7f2b3c5baa40 -1 bdev(0x55dc72896c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-13T02:24:10.883 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:24:10.871+0000 7f2b3c5baa40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-13T02:24:11.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-13T02:24:11.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:24:11.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-13T02:24:11.578 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T02:24:11.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:24:11.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-13T02:24:11.672 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T02:24:11.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_backoff_ratio=0 2026-02-13T02:24:11.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:24:11.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:24:11.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:24:11.688 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:24:11.675+0000 7f569a44da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:24:11.689 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:24:11.675+0000 7f569a44da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:24:11.690 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:24:11.679+0000 7f569a44da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:24:11.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-13T02:24:11.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:24:11.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:24:11.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:24:11.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:24:11.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:24:11.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:24:11.773 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:24:11.773 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:24:11.773 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:24:11.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:24:12.059 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:24:12.047+0000 7f569a44da40 -1 Falling back to public interface 2026-02-13T02:24:12.743 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:24:12.731+0000 7f569a44da40 -1 osd.0 0 log_to_monitors true 2026-02-13T02:24:12.866 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:24:12.866 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:24:12.867 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:24:12.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:24:12.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:24:12.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:24:12.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:24:13.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:24:13.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:24:13.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:24:13.985 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:24:13.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:24:13.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:24:14.146 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2208088566,v1:127.0.0.1:6801/2208088566] [v2:127.0.0.1:6802/2208088566,v1:127.0.0.1:6803/2208088566] exists,up 47495e47-b606-4abf-bc83-fbee61dba676 2026-02-13T02:24:14.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:24:14.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:24:14.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:24:14.147 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6027: TEST_scrub_warning: expr 5 + 5 2026-02-13T02:24:14.149 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6027: TEST_scrub_warning: seq 1 10 2026-02-13T02:24:14.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6027: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2026-02-13T02:24:14.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6029: TEST_scrub_warning: create_pool psr_pool-1 1 1 2026-02-13T02:24:14.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-1 1 1 2026-02-13T02:24:14.359 INFO:tasks.workunit.client.0.trial112.stderr:pool 'psr_pool-1' created 2026-02-13T02:24:14.369 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:24:15.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6030: TEST_scrub_warning: wait_for_clean 2026-02-13T02:24:15.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:24:15.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:24:15.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:24:15.371 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:24:15.372 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:24:15.372 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:24:15.373 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:24:15.373 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:24:15.373 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:24:15.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:24:15.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:24:15.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:24:15.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:24:15.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:24:15.490 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:24:15.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-13T02:24:15.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:24:15.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:24:15.652 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:24:15.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836482 2026-02-13T02:24:15.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836482 2026-02-13T02:24:15.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836482' 2026-02-13T02:24:15.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:24:15.726 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836482 2026-02-13T02:24:15.726 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:24:15.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:24:15.729 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836482 2026-02-13T02:24:15.729 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:24:15.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836482 2026-02-13T02:24:15.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836482' 2026-02-13T02:24:15.731 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836482 2026-02-13T02:24:15.731 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:24:15.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836481 -lt 21474836482 2026-02-13T02:24:15.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:24:16.892 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:24:16.893 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:24:17.056 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836481 -lt 21474836482 2026-02-13T02:24:17.056 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:24:18.057 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:24:18.058 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:24:18.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836482 2026-02-13T02:24:18.217 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:24:18.217 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:24:18.217 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:24:18.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:24:18.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:24:18.409 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:24:18.409 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:24:18.409 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:24:18.409 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:24:18.410 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:24:18.410 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:24:18.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:24:18.570 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:24:18.571 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:24:18.571 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:24:18.769 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:24:18.769 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:24:18.769 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:24:18.769 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6031: TEST_scrub_warning: '[' 1 = 1 ']' 2026-02-13T02:24:18.770 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6033: TEST_scrub_warning: ceph osd pool set psr_pool-1 scrub_max_interval 1209600 2026-02-13T02:24:18.971 INFO:tasks.workunit.client.0.trial112.stderr:set pool 1 scrub_max_interval to 1209600 2026-02-13T02:24:18.985 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: expr 5 + 1 2026-02-13T02:24:18.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: '[' 1 = 6 ']' 2026-02-13T02:24:18.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6027: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2026-02-13T02:24:18.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6029: TEST_scrub_warning: create_pool psr_pool-2 1 1 2026-02-13T02:24:18.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-2 1 1 2026-02-13T02:24:19.178 INFO:tasks.workunit.client.0.trial112.stderr:pool 'psr_pool-2' created 2026-02-13T02:24:19.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:24:20.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6030: TEST_scrub_warning: wait_for_clean 2026-02-13T02:24:20.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:24:20.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:24:20.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:24:20.193 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:24:20.194 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:24:20.194 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:24:20.195 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:24:20.195 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:24:20.195 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:24:20.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:24:20.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:24:20.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:24:20.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:24:20.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:24:20.312 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:24:20.474 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-13T02:24:20.474 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:24:20.474 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:24:20.475 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:24:20.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836484 2026-02-13T02:24:20.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836484 2026-02-13T02:24:20.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484' 2026-02-13T02:24:20.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:24:20.550 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836484 2026-02-13T02:24:20.550 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:24:20.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:24:20.552 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836484 2026-02-13T02:24:20.552 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:24:20.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836484 2026-02-13T02:24:20.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-13T02:24:20.554 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836484 2026-02-13T02:24:20.555 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:24:20.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836484 2026-02-13T02:24:20.718 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:24:21.718 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:24:21.719 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:24:21.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836484 2026-02-13T02:24:21.883 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:24:21.884 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:24:21.884 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:24:22.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 2 == 0 2026-02-13T02:24:22.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:24:22.079 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:24:22.080 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:24:22.080 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:24:22.080 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:24:22.080 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:24:22.080 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:24:22.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=2 2026-02-13T02:24:22.236 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:24:22.237 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:24:22.237 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:24:22.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 2 = 2 2026-02-13T02:24:22.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:24:22.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:24:22.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6031: TEST_scrub_warning: '[' 2 = 1 ']' 2026-02-13T02:24:22.424 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: expr 5 + 1 2026-02-13T02:24:22.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: '[' 2 = 6 ']' 2026-02-13T02:24:22.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6027: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2026-02-13T02:24:22.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6029: TEST_scrub_warning: create_pool psr_pool-3 1 1 2026-02-13T02:24:22.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-3 1 1 2026-02-13T02:24:22.628 INFO:tasks.workunit.client.0.trial112.stderr:pool 'psr_pool-3' created 2026-02-13T02:24:22.642 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:24:23.643 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6030: TEST_scrub_warning: wait_for_clean 2026-02-13T02:24:23.644 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:24:23.644 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:24:23.644 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:24:23.644 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:24:23.645 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:24:23.645 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:24:23.646 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:24:23.646 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:24:23.646 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:24:23.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:24:23.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:24:23.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:24:23.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:24:23.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:24:23.758 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:24:23.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-13T02:24:23.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:24:23.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:24:23.924 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:24:23.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836485 2026-02-13T02:24:23.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836485 2026-02-13T02:24:23.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485' 2026-02-13T02:24:23.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:24:23.998 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836485 2026-02-13T02:24:23.998 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:24:23.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:24:24.000 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836485 2026-02-13T02:24:24.000 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:24:24.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836485 2026-02-13T02:24:24.002 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836485 2026-02-13T02:24:24.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-13T02:24:24.003 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:24:24.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836485 2026-02-13T02:24:24.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:24:25.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:24:25.166 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:24:25.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836485 2026-02-13T02:24:25.331 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:24:25.331 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:24:25.331 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:24:25.523 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 3 == 0 2026-02-13T02:24:25.523 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:24:25.524 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:24:25.524 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:24:25.524 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:24:25.524 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:24:25.525 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:24:25.525 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:24:25.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=3 2026-02-13T02:24:25.683 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:24:25.684 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:24:25.684 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:24:25.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 3 = 3 2026-02-13T02:24:25.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:24:25.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:24:25.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6031: TEST_scrub_warning: '[' 3 = 1 ']' 2026-02-13T02:24:25.875 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: expr 5 + 1 2026-02-13T02:24:25.876 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: '[' 3 = 6 ']' 2026-02-13T02:24:25.876 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6027: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2026-02-13T02:24:25.876 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6029: TEST_scrub_warning: create_pool psr_pool-4 1 1 2026-02-13T02:24:25.876 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-4 1 1 2026-02-13T02:24:26.082 INFO:tasks.workunit.client.0.trial112.stderr:pool 'psr_pool-4' created 2026-02-13T02:24:26.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:24:27.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6030: TEST_scrub_warning: wait_for_clean 2026-02-13T02:24:27.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:24:27.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:24:27.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:24:27.095 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:24:27.096 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:24:27.096 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:24:27.097 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:24:27.097 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:24:27.097 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:24:27.218 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:24:27.218 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:24:27.218 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:24:27.218 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:24:27.219 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:24:27.219 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:24:27.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-13T02:24:27.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:24:27.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:24:27.378 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:24:27.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836487 2026-02-13T02:24:27.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836487 2026-02-13T02:24:27.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487' 2026-02-13T02:24:27.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:24:27.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836487 2026-02-13T02:24:27.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:24:27.456 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:24:27.457 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836487 2026-02-13T02:24:27.457 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:24:27.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836487 2026-02-13T02:24:27.459 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836487 2026-02-13T02:24:27.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2026-02-13T02:24:27.459 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:24:27.619 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836487 2026-02-13T02:24:27.620 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:24:28.620 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:24:28.621 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:24:28.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836487 2026-02-13T02:24:28.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:24:29.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:24:29.789 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:24:29.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836487 2026-02-13T02:24:29.953 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:24:29.954 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:24:29.954 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:24:30.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-13T02:24:30.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:24:30.148 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:24:30.149 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:24:30.149 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:24:30.149 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:24:30.149 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:24:30.149 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:24:30.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-13T02:24:30.305 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:24:30.305 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:24:30.306 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:24:30.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-13T02:24:30.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:24:30.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:24:30.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6031: TEST_scrub_warning: '[' 4 = 1 ']' 2026-02-13T02:24:30.501 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: expr 5 + 1 2026-02-13T02:24:30.502 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: '[' 4 = 6 ']' 2026-02-13T02:24:30.502 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6027: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2026-02-13T02:24:30.502 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6029: TEST_scrub_warning: create_pool psr_pool-5 1 1 2026-02-13T02:24:30.502 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-5 1 1 2026-02-13T02:24:30.710 INFO:tasks.workunit.client.0.trial112.stderr:pool 'psr_pool-5' created 2026-02-13T02:24:30.723 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:24:31.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6030: TEST_scrub_warning: wait_for_clean 2026-02-13T02:24:31.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:24:31.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:24:31.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:24:31.726 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:24:31.727 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:24:31.727 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:24:31.727 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:24:31.727 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:24:31.728 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:24:31.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:24:31.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:24:31.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:24:31.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:24:31.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:24:31.846 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:24:32.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-13T02:24:32.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:24:32.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:24:32.012 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:24:32.086 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836489 2026-02-13T02:24:32.086 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836489 2026-02-13T02:24:32.086 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489' 2026-02-13T02:24:32.086 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:24:32.087 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836489 2026-02-13T02:24:32.087 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:24:32.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:24:32.089 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836489 2026-02-13T02:24:32.089 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:24:32.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836489 2026-02-13T02:24:32.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2026-02-13T02:24:32.092 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836489 2026-02-13T02:24:32.092 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:24:32.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836489 2026-02-13T02:24:32.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:24:33.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:24:33.265 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:24:33.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836489 2026-02-13T02:24:33.423 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:24:33.424 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:24:33.424 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:24:33.618 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-13T02:24:33.618 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:24:33.619 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:24:33.619 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:24:33.620 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:24:33.620 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:24:33.620 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:24:33.620 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:24:33.773 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-13T02:24:33.774 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:24:33.775 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:24:33.775 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:24:33.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-13T02:24:33.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:24:33.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:24:33.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6031: TEST_scrub_warning: '[' 5 = 1 ']' 2026-02-13T02:24:33.973 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: expr 5 + 1 2026-02-13T02:24:33.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: '[' 5 = 6 ']' 2026-02-13T02:24:33.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6027: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2026-02-13T02:24:33.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6029: TEST_scrub_warning: create_pool psr_pool-6 1 1 2026-02-13T02:24:33.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-6 1 1 2026-02-13T02:24:34.186 INFO:tasks.workunit.client.0.trial112.stderr:pool 'psr_pool-6' created 2026-02-13T02:24:34.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:24:35.199 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6030: TEST_scrub_warning: wait_for_clean 2026-02-13T02:24:35.199 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:24:35.199 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:24:35.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:24:35.200 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:24:35.201 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:24:35.201 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:24:35.201 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:24:35.202 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:24:35.202 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:24:35.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:24:35.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:24:35.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:24:35.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:24:35.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:24:35.321 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:24:35.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-13T02:24:35.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:24:35.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:24:35.484 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:24:35.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836491 2026-02-13T02:24:35.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836491 2026-02-13T02:24:35.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491' 2026-02-13T02:24:35.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:24:35.561 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836491 2026-02-13T02:24:35.561 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:24:35.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:24:35.563 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836491 2026-02-13T02:24:35.563 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:24:35.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836491 2026-02-13T02:24:35.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2026-02-13T02:24:35.565 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836491 2026-02-13T02:24:35.566 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:24:35.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836490 -lt 21474836491 2026-02-13T02:24:35.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:24:36.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:24:36.743 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:24:36.904 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836490 -lt 21474836491 2026-02-13T02:24:36.905 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:24:37.906 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:24:37.907 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:24:38.066 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836491 2026-02-13T02:24:38.067 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:24:38.067 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:24:38.068 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:24:38.258 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-13T02:24:38.258 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:24:38.259 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:24:38.259 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:24:38.259 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:24:38.259 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:24:38.259 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:24:38.259 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:24:38.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-13T02:24:38.416 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:24:38.416 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:24:38.416 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:24:38.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-13T02:24:38.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:24:38.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:24:38.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6031: TEST_scrub_warning: '[' 6 = 1 ']' 2026-02-13T02:24:38.606 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: expr 5 + 1 2026-02-13T02:24:38.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: '[' 6 = 6 ']' 2026-02-13T02:24:38.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6037: TEST_scrub_warning: ceph osd pool set psr_pool-6 deep_scrub_interval 1209600 2026-02-13T02:24:38.807 INFO:tasks.workunit.client.0.trial112.stderr:set pool 6 deep_scrub_interval to 1209600 2026-02-13T02:24:38.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6027: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2026-02-13T02:24:38.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6029: TEST_scrub_warning: create_pool psr_pool-7 1 1 2026-02-13T02:24:38.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-7 1 1 2026-02-13T02:24:39.010 INFO:tasks.workunit.client.0.trial112.stderr:pool 'psr_pool-7' created 2026-02-13T02:24:39.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:24:40.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6030: TEST_scrub_warning: wait_for_clean 2026-02-13T02:24:40.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:24:40.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:24:40.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:24:40.023 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:24:40.023 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:24:40.024 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:24:40.024 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:24:40.024 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:24:40.024 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:24:40.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:24:40.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:24:40.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:24:40.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:24:40.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:24:40.140 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:24:40.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-13T02:24:40.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:24:40.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:24:40.308 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:24:40.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836493 2026-02-13T02:24:40.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836493 2026-02-13T02:24:40.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493' 2026-02-13T02:24:40.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:24:40.385 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836493 2026-02-13T02:24:40.385 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:24:40.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:24:40.387 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836493 2026-02-13T02:24:40.388 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:24:40.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836493 2026-02-13T02:24:40.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2026-02-13T02:24:40.390 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836493 2026-02-13T02:24:40.390 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:24:40.556 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836493 2026-02-13T02:24:40.556 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:24:41.558 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:24:41.558 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:24:41.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836493 -lt 21474836493 2026-02-13T02:24:41.721 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:24:41.722 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:24:41.722 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:24:41.913 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 7 == 0 2026-02-13T02:24:41.913 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:24:41.914 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:24:41.914 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:24:41.914 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:24:41.914 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:24:41.914 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:24:41.914 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:24:42.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=7 2026-02-13T02:24:42.071 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:24:42.071 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:24:42.072 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:24:42.267 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 7 = 7 2026-02-13T02:24:42.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:24:42.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:24:42.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6031: TEST_scrub_warning: '[' 7 = 1 ']' 2026-02-13T02:24:42.268 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: expr 5 + 1 2026-02-13T02:24:42.269 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: '[' 7 = 6 ']' 2026-02-13T02:24:42.270 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6027: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2026-02-13T02:24:42.270 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6029: TEST_scrub_warning: create_pool psr_pool-8 1 1 2026-02-13T02:24:42.270 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-8 1 1 2026-02-13T02:24:42.479 INFO:tasks.workunit.client.0.trial112.stderr:pool 'psr_pool-8' created 2026-02-13T02:24:42.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:24:43.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6030: TEST_scrub_warning: wait_for_clean 2026-02-13T02:24:43.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:24:43.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:24:43.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:24:43.493 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:24:43.494 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:24:43.494 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:24:43.495 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:24:43.495 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:24:43.495 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:24:43.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:24:43.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:24:43.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:24:43.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:24:43.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:24:43.616 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:24:43.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-13T02:24:43.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:24:43.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:24:43.777 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:24:43.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836494 2026-02-13T02:24:43.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836494 2026-02-13T02:24:43.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836494' 2026-02-13T02:24:43.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:24:43.850 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836494 2026-02-13T02:24:43.850 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:24:43.852 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:24:43.853 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836494 2026-02-13T02:24:43.853 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:24:43.854 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836494 2026-02-13T02:24:43.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2026-02-13T02:24:43.855 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836494 2026-02-13T02:24:43.855 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:24:44.021 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836493 -lt 21474836494 2026-02-13T02:24:44.021 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:24:45.023 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:24:45.023 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:24:45.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836495 -lt 21474836494 2026-02-13T02:24:45.193 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:24:45.194 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:24:45.194 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:24:45.394 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 8 == 0 2026-02-13T02:24:45.394 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:24:45.395 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:24:45.395 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:24:45.395 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:24:45.395 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:24:45.395 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:24:45.395 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:24:45.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=8 2026-02-13T02:24:45.561 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:24:45.561 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:24:45.562 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:24:45.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 8 = 8 2026-02-13T02:24:45.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:24:45.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:24:45.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6031: TEST_scrub_warning: '[' 8 = 1 ']' 2026-02-13T02:24:45.760 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: expr 5 + 1 2026-02-13T02:24:45.760 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: '[' 8 = 6 ']' 2026-02-13T02:24:45.760 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6027: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2026-02-13T02:24:45.760 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6029: TEST_scrub_warning: create_pool psr_pool-9 1 1 2026-02-13T02:24:45.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-9 1 1 2026-02-13T02:24:45.969 INFO:tasks.workunit.client.0.trial112.stderr:pool 'psr_pool-9' created 2026-02-13T02:24:45.980 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:24:46.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6030: TEST_scrub_warning: wait_for_clean 2026-02-13T02:24:46.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:24:46.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:24:46.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:24:46.982 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:24:46.983 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:24:46.983 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:24:46.984 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:24:46.984 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:24:46.984 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:24:47.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:24:47.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:24:47.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:24:47.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:24:47.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:24:47.109 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:24:47.270 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-13T02:24:47.270 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:24:47.270 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:24:47.271 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:24:47.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836496 2026-02-13T02:24:47.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836496 2026-02-13T02:24:47.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836496' 2026-02-13T02:24:47.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:24:47.346 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836496 2026-02-13T02:24:47.346 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:24:47.347 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:24:47.348 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836496 2026-02-13T02:24:47.348 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:24:47.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836496 2026-02-13T02:24:47.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2026-02-13T02:24:47.351 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836496 2026-02-13T02:24:47.351 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:24:47.510 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836495 -lt 21474836496 2026-02-13T02:24:47.510 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:24:48.512 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:24:48.512 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:24:48.680 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836495 -lt 21474836496 2026-02-13T02:24:48.680 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:24:49.681 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:24:49.682 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:24:49.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836496 -lt 21474836496 2026-02-13T02:24:49.847 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:24:49.848 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:24:49.848 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:24:50.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 9 == 0 2026-02-13T02:24:50.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:24:50.041 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:24:50.041 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:24:50.041 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:24:50.041 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:24:50.041 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:24:50.041 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:24:50.195 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=9 2026-02-13T02:24:50.196 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:24:50.196 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:24:50.196 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:24:50.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 9 = 9 2026-02-13T02:24:50.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:24:50.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:24:50.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6031: TEST_scrub_warning: '[' 9 = 1 ']' 2026-02-13T02:24:50.388 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: expr 5 + 1 2026-02-13T02:24:50.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: '[' 9 = 6 ']' 2026-02-13T02:24:50.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6027: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2026-02-13T02:24:50.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6029: TEST_scrub_warning: create_pool psr_pool-10 1 1 2026-02-13T02:24:50.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-10 1 1 2026-02-13T02:24:50.631 INFO:tasks.workunit.client.0.trial112.stderr:pool 'psr_pool-10' created 2026-02-13T02:24:50.643 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:24:51.645 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6030: TEST_scrub_warning: wait_for_clean 2026-02-13T02:24:51.645 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:24:51.645 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:24:51.645 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:24:51.645 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:24:51.646 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:24:51.646 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:24:51.647 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:24:51.647 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:24:51.647 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:24:51.766 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:24:51.766 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:24:51.766 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:24:51.767 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:24:51.767 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:24:51.767 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:24:51.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-13T02:24:51.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:24:51.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:24:51.931 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:24:52.004 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836498 2026-02-13T02:24:52.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836498 2026-02-13T02:24:52.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836498' 2026-02-13T02:24:52.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:24:52.005 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836498 2026-02-13T02:24:52.006 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:24:52.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:24:52.008 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836498 2026-02-13T02:24:52.008 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:24:52.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836498 2026-02-13T02:24:52.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2026-02-13T02:24:52.010 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836498 2026-02-13T02:24:52.011 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:24:52.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836497 -lt 21474836498 2026-02-13T02:24:52.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:24:53.170 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:24:53.170 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:24:53.336 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836498 -lt 21474836498 2026-02-13T02:24:53.337 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:24:53.337 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:24:53.338 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:24:53.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 10 == 0 2026-02-13T02:24:53.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:24:53.534 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:24:53.534 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:24:53.534 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:24:53.534 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:24:53.534 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:24:53.534 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:24:53.694 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=10 2026-02-13T02:24:53.695 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:24:53.696 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:24:53.696 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:24:53.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 10 = 10 2026-02-13T02:24:53.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:24:53.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:24:53.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6031: TEST_scrub_warning: '[' 10 = 1 ']' 2026-02-13T02:24:53.891 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: expr 5 + 1 2026-02-13T02:24:53.892 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: '[' 10 = 6 ']' 2026-02-13T02:24:53.892 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6042: TEST_scrub_warning: local primary=0 2026-02-13T02:24:53.892 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6044: TEST_scrub_warning: ceph osd set noscrub 2026-02-13T02:24:54.100 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:24:54.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6045: TEST_scrub_warning: ceph osd set nodeep-scrub 2026-02-13T02:24:54.304 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:24:54.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6046: TEST_scrub_warning: ceph config set global osd_scrub_interval_randomize_ratio 0 2026-02-13T02:24:54.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6047: TEST_scrub_warning: ceph config set global osd_deep_scrub_randomize_ratio 0 2026-02-13T02:24:54.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6048: TEST_scrub_warning: ceph config set global osd_scrub_max_interval 604800 2026-02-13T02:24:54.845 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6049: TEST_scrub_warning: ceph config set global osd_deep_scrub_interval 604800 2026-02-13T02:24:55.010 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6052: TEST_scrub_warning: seq 1 5 2026-02-13T02:24:55.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6052: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2026-02-13T02:24:55.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6054: TEST_scrub_warning: '[' 1 = 1 ']' 2026-02-13T02:24:55.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6056: TEST_scrub_warning: overdue_seconds=1900800 2026-02-13T02:24:55.012 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6060: TEST_scrub_warning: expr 1900800 + 100 2026-02-13T02:24:55.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6060: TEST_scrub_warning: ceph tell 1.0 schedule-scrub 1900900 2026-02-13T02:24:55.082 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:24:55.083 INFO:tasks.workunit.client.0.trial112.stdout: "deep": false, 2026-02-13T02:24:55.083 INFO:tasks.workunit.client.0.trial112.stdout: "must": false, 2026-02-13T02:24:55.083 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "2026-01-22T02:23:15.076176+0000" 2026-02-13T02:24:55.083 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:24:55.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6052: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2026-02-13T02:24:55.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6054: TEST_scrub_warning: '[' 2 = 1 ']' 2026-02-13T02:24:55.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6058: TEST_scrub_warning: overdue_seconds=993600 2026-02-13T02:24:55.091 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6060: TEST_scrub_warning: expr 993600 + 200 2026-02-13T02:24:55.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6060: TEST_scrub_warning: ceph tell 2.0 schedule-scrub 993800 2026-02-13T02:24:55.158 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:24:55.158 INFO:tasks.workunit.client.0.trial112.stdout: "deep": false, 2026-02-13T02:24:55.158 INFO:tasks.workunit.client.0.trial112.stdout: "must": false, 2026-02-13T02:24:55.158 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "2026-02-01T14:21:35.151880+0000" 2026-02-13T02:24:55.159 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:24:55.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6052: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2026-02-13T02:24:55.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6054: TEST_scrub_warning: '[' 3 = 1 ']' 2026-02-13T02:24:55.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6058: TEST_scrub_warning: overdue_seconds=993600 2026-02-13T02:24:55.166 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6060: TEST_scrub_warning: expr 993600 + 300 2026-02-13T02:24:55.167 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6060: TEST_scrub_warning: ceph tell 3.0 schedule-scrub 993900 2026-02-13T02:24:55.234 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:24:55.234 INFO:tasks.workunit.client.0.trial112.stdout: "deep": false, 2026-02-13T02:24:55.234 INFO:tasks.workunit.client.0.trial112.stdout: "must": false, 2026-02-13T02:24:55.234 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "2026-02-01T14:19:55.227650+0000" 2026-02-13T02:24:55.234 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:24:55.242 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6052: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2026-02-13T02:24:55.242 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6054: TEST_scrub_warning: '[' 4 = 1 ']' 2026-02-13T02:24:55.242 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6058: TEST_scrub_warning: overdue_seconds=993600 2026-02-13T02:24:55.242 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6060: TEST_scrub_warning: expr 993600 + 400 2026-02-13T02:24:55.243 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6060: TEST_scrub_warning: ceph tell 4.0 schedule-scrub 994000 2026-02-13T02:24:55.310 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:24:55.310 INFO:tasks.workunit.client.0.trial112.stdout: "deep": false, 2026-02-13T02:24:55.310 INFO:tasks.workunit.client.0.trial112.stdout: "must": false, 2026-02-13T02:24:55.311 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "2026-02-01T14:18:15.303964+0000" 2026-02-13T02:24:55.311 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:24:55.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6052: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2026-02-13T02:24:55.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6054: TEST_scrub_warning: '[' 5 = 1 ']' 2026-02-13T02:24:55.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6058: TEST_scrub_warning: overdue_seconds=993600 2026-02-13T02:24:55.318 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6060: TEST_scrub_warning: expr 993600 + 500 2026-02-13T02:24:55.319 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6060: TEST_scrub_warning: ceph tell 5.0 schedule-scrub 994100 2026-02-13T02:24:55.386 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:24:55.386 INFO:tasks.workunit.client.0.trial112.stdout: "deep": false, 2026-02-13T02:24:55.387 INFO:tasks.workunit.client.0.trial112.stdout: "must": false, 2026-02-13T02:24:55.387 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "2026-02-01T14:16:35.380249+0000" 2026-02-13T02:24:55.387 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:24:55.398 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6063: TEST_scrub_warning: expr 5 + 1 2026-02-13T02:24:55.400 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6063: TEST_scrub_warning: expr 5 + 5 2026-02-13T02:24:55.401 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6063: TEST_scrub_warning: seq 6 10 2026-02-13T02:24:55.402 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6063: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2026-02-13T02:24:55.402 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6065: TEST_scrub_warning: expr 5 + 1 2026-02-13T02:24:55.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6065: TEST_scrub_warning: '[' 6 = 6 ']' 2026-02-13T02:24:55.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6067: TEST_scrub_warning: overdue_seconds=1900800 2026-02-13T02:24:55.405 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6071: TEST_scrub_warning: expr 1900800 + 600 2026-02-13T02:24:55.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6071: TEST_scrub_warning: ceph tell 6.0 schedule-deep-scrub 1901400 2026-02-13T02:24:55.471 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:24:55.472 INFO:tasks.workunit.client.0.trial112.stdout: "deep": true, 2026-02-13T02:24:55.472 INFO:tasks.workunit.client.0.trial112.stdout: "must": false, 2026-02-13T02:24:55.472 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "2026-01-22T02:14:55.465162+0000" 2026-02-13T02:24:55.472 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:24:55.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6063: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2026-02-13T02:24:55.479 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6065: TEST_scrub_warning: expr 5 + 1 2026-02-13T02:24:55.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6065: TEST_scrub_warning: '[' 7 = 6 ']' 2026-02-13T02:24:55.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6069: TEST_scrub_warning: overdue_seconds=993600 2026-02-13T02:24:55.481 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6071: TEST_scrub_warning: expr 993600 + 700 2026-02-13T02:24:55.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6071: TEST_scrub_warning: ceph tell 7.0 schedule-deep-scrub 994300 2026-02-13T02:24:55.547 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:24:55.547 INFO:tasks.workunit.client.0.trial112.stdout: "deep": true, 2026-02-13T02:24:55.547 INFO:tasks.workunit.client.0.trial112.stdout: "must": false, 2026-02-13T02:24:55.547 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "2026-02-01T14:13:15.540394+0000" 2026-02-13T02:24:55.547 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:24:55.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6063: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2026-02-13T02:24:55.554 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6065: TEST_scrub_warning: expr 5 + 1 2026-02-13T02:24:55.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6065: TEST_scrub_warning: '[' 8 = 6 ']' 2026-02-13T02:24:55.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6069: TEST_scrub_warning: overdue_seconds=993600 2026-02-13T02:24:55.556 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6071: TEST_scrub_warning: expr 993600 + 800 2026-02-13T02:24:55.557 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6071: TEST_scrub_warning: ceph tell 8.0 schedule-deep-scrub 994400 2026-02-13T02:24:55.626 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:24:55.626 INFO:tasks.workunit.client.0.trial112.stdout: "deep": true, 2026-02-13T02:24:55.626 INFO:tasks.workunit.client.0.trial112.stdout: "must": false, 2026-02-13T02:24:55.626 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "2026-02-01T14:11:35.619577+0000" 2026-02-13T02:24:55.626 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:24:55.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6063: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2026-02-13T02:24:55.634 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6065: TEST_scrub_warning: expr 5 + 1 2026-02-13T02:24:55.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6065: TEST_scrub_warning: '[' 9 = 6 ']' 2026-02-13T02:24:55.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6069: TEST_scrub_warning: overdue_seconds=993600 2026-02-13T02:24:55.636 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6071: TEST_scrub_warning: expr 993600 + 900 2026-02-13T02:24:55.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6071: TEST_scrub_warning: ceph tell 9.0 schedule-deep-scrub 994500 2026-02-13T02:24:55.702 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:24:55.703 INFO:tasks.workunit.client.0.trial112.stdout: "deep": true, 2026-02-13T02:24:55.703 INFO:tasks.workunit.client.0.trial112.stdout: "must": false, 2026-02-13T02:24:55.703 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "2026-02-01T14:09:55.696235+0000" 2026-02-13T02:24:55.703 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:24:55.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6063: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2026-02-13T02:24:55.710 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6065: TEST_scrub_warning: expr 5 + 1 2026-02-13T02:24:55.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6065: TEST_scrub_warning: '[' 10 = 6 ']' 2026-02-13T02:24:55.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6069: TEST_scrub_warning: overdue_seconds=993600 2026-02-13T02:24:55.712 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6071: TEST_scrub_warning: expr 993600 + 1000 2026-02-13T02:24:55.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6071: TEST_scrub_warning: ceph tell 10.0 schedule-deep-scrub 994600 2026-02-13T02:24:55.779 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:24:55.779 INFO:tasks.workunit.client.0.trial112.stdout: "deep": true, 2026-02-13T02:24:55.779 INFO:tasks.workunit.client.0.trial112.stdout: "must": false, 2026-02-13T02:24:55.780 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "2026-02-01T14:08:15.772998+0000" 2026-02-13T02:24:55.780 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:24:55.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6073: TEST_scrub_warning: flush_pg_stats 2026-02-13T02:24:55.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:24:55.787 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:24:55.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-13T02:24:55.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:24:55.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:24:55.965 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:24:56.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836500 2026-02-13T02:24:56.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836500 2026-02-13T02:24:56.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836500' 2026-02-13T02:24:56.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:24:56.041 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836500 2026-02-13T02:24:56.041 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:24:56.042 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:24:56.043 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836500 2026-02-13T02:24:56.043 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:24:56.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836500 2026-02-13T02:24:56.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2026-02-13T02:24:56.045 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836500 2026-02-13T02:24:56.046 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:24:56.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836499 -lt 21474836500 2026-02-13T02:24:56.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:24:57.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:24:57.206 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:24:57.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836500 -lt 21474836500 2026-02-13T02:24:57.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6075: TEST_scrub_warning: ceph health 2026-02-13T02:24:57.543 INFO:tasks.workunit.client.0.trial112.stdout:HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 5 pgs not deep-scrubbed in time; 5 pgs not scrubbed in time; 10 pool(s) have no replicas configured; OSD count 1 < osd_pool_default_size 3 2026-02-13T02:24:57.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6076: TEST_scrub_warning: ceph health detail 2026-02-13T02:24:57.752 INFO:tasks.workunit.client.0.trial112.stdout:HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 5 pgs not deep-scrubbed in time; 5 pgs not scrubbed in time; 10 pool(s) have no replicas configured; OSD count 1 < osd_pool_default_size 3 2026-02-13T02:24:57.752 INFO:tasks.workunit.client.0.trial112.stdout:[WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2026-02-13T02:24:57.752 INFO:tasks.workunit.client.0.trial112.stdout:[WRN] PG_NOT_DEEP_SCRUBBED: 5 pgs not deep-scrubbed in time 2026-02-13T02:24:57.752 INFO:tasks.workunit.client.0.trial112.stdout: pg 10.0 not deep-scrubbed since 2026-02-01T14:08:15.772998+0000 2026-02-13T02:24:57.752 INFO:tasks.workunit.client.0.trial112.stdout: pg 9.0 not deep-scrubbed since 2026-02-01T14:09:55.696235+0000 2026-02-13T02:24:57.752 INFO:tasks.workunit.client.0.trial112.stdout: pg 8.0 not deep-scrubbed since 2026-02-01T14:11:35.619577+0000 2026-02-13T02:24:57.752 INFO:tasks.workunit.client.0.trial112.stdout: pg 7.0 not deep-scrubbed since 2026-02-01T14:13:15.540394+0000 2026-02-13T02:24:57.752 INFO:tasks.workunit.client.0.trial112.stdout: pg 6.0 not deep-scrubbed since 2026-01-22T02:14:55.465162+0000 2026-02-13T02:24:57.752 INFO:tasks.workunit.client.0.trial112.stdout:[WRN] PG_NOT_SCRUBBED: 5 pgs not scrubbed in time 2026-02-13T02:24:57.753 INFO:tasks.workunit.client.0.trial112.stdout: pg 5.0 not scrubbed since 2026-02-01T14:16:35.380249+0000 2026-02-13T02:24:57.753 INFO:tasks.workunit.client.0.trial112.stdout: pg 4.0 not scrubbed since 2026-02-01T14:18:15.303964+0000 2026-02-13T02:24:57.753 INFO:tasks.workunit.client.0.trial112.stdout: pg 3.0 not scrubbed since 2026-02-01T14:19:55.227650+0000 2026-02-13T02:24:57.753 INFO:tasks.workunit.client.0.trial112.stdout: pg 2.0 not scrubbed since 2026-02-01T14:21:35.151880+0000 2026-02-13T02:24:57.753 INFO:tasks.workunit.client.0.trial112.stdout: pg 1.0 not scrubbed since 2026-01-22T02:23:15.076176+0000 2026-02-13T02:24:57.753 INFO:tasks.workunit.client.0.trial112.stdout:[WRN] POOL_NO_REDUNDANCY: 10 pool(s) have no replicas configured 2026-02-13T02:24:57.753 INFO:tasks.workunit.client.0.trial112.stdout: pool 'psr_pool-1' has no replicas configured 2026-02-13T02:24:57.753 INFO:tasks.workunit.client.0.trial112.stdout: pool 'psr_pool-2' has no replicas configured 2026-02-13T02:24:57.753 INFO:tasks.workunit.client.0.trial112.stdout: pool 'psr_pool-3' has no replicas configured 2026-02-13T02:24:57.753 INFO:tasks.workunit.client.0.trial112.stdout: pool 'psr_pool-4' has no replicas configured 2026-02-13T02:24:57.753 INFO:tasks.workunit.client.0.trial112.stdout: pool 'psr_pool-5' has no replicas configured 2026-02-13T02:24:57.753 INFO:tasks.workunit.client.0.trial112.stdout: pool 'psr_pool-6' has no replicas configured 2026-02-13T02:24:57.753 INFO:tasks.workunit.client.0.trial112.stdout: pool 'psr_pool-7' has no replicas configured 2026-02-13T02:24:57.754 INFO:tasks.workunit.client.0.trial112.stdout: pool 'psr_pool-8' has no replicas configured 2026-02-13T02:24:57.754 INFO:tasks.workunit.client.0.trial112.stdout: pool 'psr_pool-9' has no replicas configured 2026-02-13T02:24:57.754 INFO:tasks.workunit.client.0.trial112.stdout: pool 'psr_pool-10' has no replicas configured 2026-02-13T02:24:57.754 INFO:tasks.workunit.client.0.trial112.stdout:[WRN] TOO_FEW_OSDS: OSD count 1 < osd_pool_default_size 3 2026-02-13T02:24:57.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6077: TEST_scrub_warning: ceph health 2026-02-13T02:24:57.762 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6077: TEST_scrub_warning: grep -q ' pgs not deep-scrubbed in time' 2026-02-13T02:24:57.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6078: TEST_scrub_warning: ceph health 2026-02-13T02:24:57.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6078: TEST_scrub_warning: grep -q ' pgs not scrubbed in time' 2026-02-13T02:24:58.147 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6083: TEST_scrub_warning: ceph health detail 2026-02-13T02:24:58.147 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6083: TEST_scrub_warning: grep 'not scrubbed since' 2026-02-13T02:24:58.147 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6083: TEST_scrub_warning: wc -l 2026-02-13T02:24:58.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6083: TEST_scrub_warning: COUNT=5 2026-02-13T02:24:58.345 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6085: TEST_scrub_warning: expr 5+5 2026-02-13T02:24:58.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6085: TEST_scrub_warning: (( 5 != 5 && 5 != 5+5 )) 2026-02-13T02:24:58.347 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6089: TEST_scrub_warning: ceph health detail 2026-02-13T02:24:58.347 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6089: TEST_scrub_warning: grep 'not deep-scrubbed since' 2026-02-13T02:24:58.347 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6089: TEST_scrub_warning: wc -l 2026-02-13T02:24:58.548 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6089: TEST_scrub_warning: COUNT=5 2026-02-13T02:24:58.548 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6090: TEST_scrub_warning: '[' 5 '!=' 5 ']' 2026-02-13T02:24:58.548 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-13T02:24:58.548 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:24:58.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:24:58.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:24:58.549 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:24:58.549 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:24:58.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:24:58.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:24:58.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:24:58.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:24:58.658 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:24:58.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:24:58.660 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:24:58.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:24:58.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:24:58.663 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:24:58.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:24:58.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:24:58.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:24:58.664 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:24:58.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:24:58.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:24:58.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:24:58.675 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:24:58.675 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:24:58.676 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:24:58.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:24:58.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:24:58.678 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:24:58.678 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-13T02:24:58.678 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-13T02:24:58.678 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-13T02:24:58.678 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-13T02:24:58.678 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:24:58.678 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:24:58.678 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:24:58.679 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:24:58.679 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:24:58.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:24:58.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:24:58.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:24:58.681 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:24:58.682 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:24:58.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:24:58.684 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:24:58.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:24:58.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:24:58.685 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:24:58.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:24:58.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:24:58.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:24:58.687 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:24:58.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:24:58.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:24:58.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:24:58.691 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:24:58.691 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:24:58.691 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:24:58.692 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:24:58.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:24:58.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:24:58.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-13T02:24:58.695 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T02:24:58.695 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:24:58.695 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:24:58.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.35207 2026-02-13T02:24:58.697 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T02:24:58.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-13T02:24:58.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T02:24:58.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-13T02:24:58.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_skip_non_repair_during_recovery td/osd-scrub-repair 2026-02-13T02:24:58.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:160: TEST_skip_non_repair_during_recovery: local dir=td/osd-scrub-repair 2026-02-13T02:24:58.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:161: TEST_skip_non_repair_during_recovery: cluster_conf=(['osds_num']='2' ['pgs_in_pool']='4' ['pool_name']='nopool' ['pool_default_size']='2' ['extras']='--osd_scrub_during_recovery=false --osd_debug_pretend_recovery_active=true') 2026-02-13T02:24:58.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:161: TEST_skip_non_repair_during_recovery: local -A cluster_conf 2026-02-13T02:24:58.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: TEST_skip_non_repair_during_recovery: standard_scrub_cluster td/osd-scrub-repair cluster_conf 2026-02-13T02:24:58.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local dir=td/osd-scrub-repair 2026-02-13T02:24:58.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:230: standard_scrub_cluster: local -n args=cluster_conf 2026-02-13T02:24:58.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local OSDS=2 2026-02-13T02:24:58.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: local pg_num=4 2026-02-13T02:24:58.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:235: standard_scrub_cluster: local poolname=nopool 2026-02-13T02:24:58.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:236: standard_scrub_cluster: args['pool_name']=nopool 2026-02-13T02:24:58.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:238: standard_scrub_cluster: local pool_default_size=2 2026-02-13T02:24:58.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:239: standard_scrub_cluster: args['pool_default_size']=2 2026-02-13T02:24:58.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:241: standard_scrub_cluster: local 'extra_pars=--osd_scrub_during_recovery=false --osd_debug_pretend_recovery_active=true' 2026-02-13T02:24:58.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:242: standard_scrub_cluster: local debug_msg=dbg 2026-02-13T02:24:58.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:245: standard_scrub_cluster: local saved_echo_flag=x 2026-02-13T02:24:58.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:246: standard_scrub_cluster: set +x 2026-02-13T02:24:58.990 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 227f61b4-b86c-4e96-8ec7-ee44ffa2dea3 2026-02-13T02:24:59.092 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:24:59.123 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:24:59.111+0000 7f11f8f2ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:24:59.124 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:24:59.111+0000 7f11f8f2ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:24:59.125 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:24:59.115+0000 7f11f8f2ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:24:59.126 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:24:59.115+0000 7f11f8f2ea40 -1 bdev(0x562232aeec00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-13T02:24:59.126 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:24:59.115+0000 7f11f8f2ea40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-13T02:24:59.825 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T02:24:59.920 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T02:24:59.955 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:24:59.943+0000 7fa6a46c0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:24:59.957 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:24:59.943+0000 7fa6a46c0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:24:59.958 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:24:59.947+0000 7fa6a46c0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:00.011 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:25:00.419 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:00.407+0000 7fa6a46c0a40 -1 Falling back to public interface 2026-02-13T02:25:01.103 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:25:01.157 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:01.147+0000 7fa6a46c0a40 -1 osd.0 0 log_to_monitors true 2026-02-13T02:25:02.226 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:25:02.395 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1666918077,v1:127.0.0.1:6801/1666918077] [v2:127.0.0.1:6802/1666918077,v1:127.0.0.1:6803/1666918077] exists,up 227f61b4-b86c-4e96-8ec7-ee44ffa2dea3 2026-02-13T02:25:02.402 INFO:tasks.workunit.client.0.trial112.stdout:add osd1 9da07ccd-85a5-4bde-a7ad-21a094279038 2026-02-13T02:25:02.567 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:25:02.606 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:02.595+0000 7f5a64f70a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:02.608 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:02.595+0000 7f5a64f70a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:02.609 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:02.595+0000 7f5a64f70a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:02.609 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:02.599+0000 7f5a64f70a40 -1 bdev(0x55c393260c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-13T02:25:02.609 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:02.599+0000 7f5a64f70a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-13T02:25:03.257 INFO:tasks.workunit.client.0.trial112.stdout:adding osd1 key to auth repository 2026-02-13T02:25:03.462 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T02:25:03.493 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:03.483+0000 7f2da06f6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:03.495 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:03.483+0000 7f2da06f6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:03.496 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:03.483+0000 7f2da06f6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:03.619 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:25:03.919 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:03.907+0000 7f2da06f6a40 -1 Falling back to public interface 2026-02-13T02:25:04.513 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:04.499+0000 7f2da06f6a40 -1 osd.1 0 log_to_monitors true 2026-02-13T02:25:04.785 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:25:05.945 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:25:06.109 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1636974740,v1:127.0.0.1:6811/1636974740] [v2:127.0.0.1:6812/1636974740,v1:127.0.0.1:6813/1636974740] exists,up 9da07ccd-85a5-4bde-a7ad-21a094279038 2026-02-13T02:25:07.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:170: TEST_skip_non_repair_during_recovery: local poolname=rbd 2026-02-13T02:25:07.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:171: TEST_skip_non_repair_during_recovery: create_rbd_pool 2026-02-13T02:25:07.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-13T02:25:07.266 INFO:tasks.workunit.client.0.trial112.stderr:pool 'rbd' does not exist 2026-02-13T02:25:07.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-13T02:25:07.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-13T02:25:07.454 INFO:tasks.workunit.client.0.trial112.stderr:pool 'rbd' created 2026-02-13T02:25:07.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:25:08.465 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-13T02:25:08.748 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:172: TEST_skip_non_repair_during_recovery: wait_for_clean 2026-02-13T02:25:08.748 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:25:08.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:25:08.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:25:08.750 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:25:08.750 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:25:08.750 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:25:08.750 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:25:08.750 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:25:08.750 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:25:08.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:25:08.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:25:08.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:25:08.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:25:08.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:25:08.873 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:25:09.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:25:09.032 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:25:09.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:25:09.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:25:09.033 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:25:09.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836488 2026-02-13T02:25:09.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836488 2026-02-13T02:25:09.107 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488' 2026-02-13T02:25:09.107 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:25:09.107 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:25:09.180 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705669 2026-02-13T02:25:09.180 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705669 2026-02-13T02:25:09.180 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-38654705669' 2026-02-13T02:25:09.180 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:25:09.181 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836488 2026-02-13T02:25:09.181 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:25:09.182 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:25:09.183 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836488 2026-02-13T02:25:09.183 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:25:09.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836488 2026-02-13T02:25:09.185 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836488 2026-02-13T02:25:09.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2026-02-13T02:25:09.186 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:25:09.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836488 2026-02-13T02:25:09.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:25:09.352 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705669 2026-02-13T02:25:09.352 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:25:09.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:25:09.355 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705669 2026-02-13T02:25:09.355 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:25:09.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705669 2026-02-13T02:25:09.357 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705669 2026-02-13T02:25:09.357 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705669' 2026-02-13T02:25:09.357 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:25:09.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705669 -lt 38654705669 2026-02-13T02:25:09.529 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:25:09.530 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:25:09.530 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:25:09.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-13T02:25:09.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:25:09.728 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:25:09.728 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:25:09.728 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:25:09.728 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:25:09.729 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:25:09.729 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:25:09.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-13T02:25:09.888 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:25:09.889 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:25:09.889 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:25:10.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-13T02:25:10.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:25:10.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:25:10.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:174: TEST_skip_non_repair_during_recovery: add_something td/osd-scrub-repair rbd 2026-02-13T02:25:10.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:25:10.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2026-02-13T02:25:10.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2026-02-13T02:25:10.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:25:10.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:25:10.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:25:10.288 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:25:10.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:25:10.492 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:25:10.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:25:10.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:25:10.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-13T02:25:10.531 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:175: TEST_skip_non_repair_during_recovery: get_not_primary rbd SOMETHING 2026-02-13T02:25:10.531 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=rbd 2026-02-13T02:25:10.531 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-13T02:25:10.532 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary rbd SOMETHING 2026-02-13T02:25:10.532 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2026-02-13T02:25:10.532 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-13T02:25:10.533 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd SOMETHING 2026-02-13T02:25:10.533 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T02:25:10.690 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-13T02:25:10.691 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map rbd SOMETHING 2026-02-13T02:25:10.691 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-13T02:25:10.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:175: TEST_skip_non_repair_during_recovery: scrub_and_not_schedule td/osd-scrub-repair rbd 0 2026-02-13T02:25:10.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:214: scrub_and_not_schedule: local dir=td/osd-scrub-repair 2026-02-13T02:25:10.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: scrub_and_not_schedule: local poolname=rbd 2026-02-13T02:25:10.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: scrub_and_not_schedule: local osd=0 2026-02-13T02:25:10.857 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:221: scrub_and_not_schedule: get_pg rbd SOMETHING 2026-02-13T02:25:10.857 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=rbd 2026-02-13T02:25:10.857 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-13T02:25:10.857 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map rbd SOMETHING 2026-02-13T02:25:10.857 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T02:25:11.015 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:221: scrub_and_not_schedule: local pg=1.3 2026-02-13T02:25:11.016 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:222: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2026-02-13T02:25:11.016 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-13T02:25:11.017 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:25:11.017 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:25:11.017 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-13T02:25:11.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:222: scrub_and_not_schedule: local last_scrub=2026-02-13T02:25:07.445357+0000 2026-02-13T02:25:11.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:223: scrub_and_not_schedule: ceph tell 1.3 schedule-scrub 2026-02-13T02:25:11.238 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:25:11.239 INFO:tasks.workunit.client.0.trial112.stdout: "deep": false, 2026-02-13T02:25:11.239 INFO:tasks.workunit.client.0.trial112.stdout: "must": false, 2026-02-13T02:25:11.239 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "2026-02-11T02:25:11.232329+0000" 2026-02-13T02:25:11.239 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:25:11.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: scrub_and_not_schedule: sleep 3 2026-02-13T02:25:14.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:229: scrub_and_not_schedule: (( i=0 )) 2026-02-13T02:25:14.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:229: scrub_and_not_schedule: (( i < 3 )) 2026-02-13T02:25:14.248 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:230: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2026-02-13T02:25:14.248 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-13T02:25:14.248 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:25:14.249 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:25:14.249 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-13T02:25:14.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:230: scrub_and_not_schedule: test 2026-02-11T02:25:11.232329+0000 '>' 2026-02-13T02:25:07.445357+0000 2026-02-13T02:25:14.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: scrub_and_not_schedule: sleep 1 2026-02-13T02:25:15.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:229: scrub_and_not_schedule: (( i++ )) 2026-02-13T02:25:15.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:229: scrub_and_not_schedule: (( i < 3 )) 2026-02-13T02:25:15.412 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:230: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2026-02-13T02:25:15.412 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-13T02:25:15.412 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:25:15.413 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-13T02:25:15.413 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:25:15.575 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:230: scrub_and_not_schedule: test 2026-02-11T02:25:11.232329+0000 '>' 2026-02-13T02:25:07.445357+0000 2026-02-13T02:25:15.575 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: scrub_and_not_schedule: sleep 1 2026-02-13T02:25:16.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:229: scrub_and_not_schedule: (( i++ )) 2026-02-13T02:25:16.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:229: scrub_and_not_schedule: (( i < 3 )) 2026-02-13T02:25:16.578 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:230: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2026-02-13T02:25:16.578 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-13T02:25:16.578 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:25:16.578 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:25:16.578 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-13T02:25:16.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:230: scrub_and_not_schedule: test 2026-02-11T02:25:11.232329+0000 '>' 2026-02-13T02:25:07.445357+0000 2026-02-13T02:25:16.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: scrub_and_not_schedule: sleep 1 2026-02-13T02:25:17.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:229: scrub_and_not_schedule: (( i++ )) 2026-02-13T02:25:17.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:229: scrub_and_not_schedule: (( i < 3 )) 2026-02-13T02:25:17.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:239: scrub_and_not_schedule: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-13T02:25:17.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:25:17.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:25:17.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T02:25:17.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:25:17.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-13T02:25:17.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:25:17.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:25:17.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T02:25:17.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:25:17.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T02:25:17.744 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:25:17.744 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:25:17.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:25:17.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:25:17.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:25:18.054 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:25:18.054 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-13T02:25:18.055 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:25:18.055 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:25:18.055 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:25:18.055 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:25:18.055 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:25:18.055 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2026-02-13T02:25:18.139 INFO:tasks.workunit.client.0.trial112.stdout:_ 2026-02-13T02:25:18.139 INFO:tasks.workunit.client.0.trial112.stdout:snapset 2026-02-13T02:25:18.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-13T02:25:18.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:25:18.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:25:18.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:25:18.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:25:18.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:25:18.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:25:18.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:25:18.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:25:18.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:25:18.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:25:18.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:25:18.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:25:18.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:25:18.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:25:18.240 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:25:18.240 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:25:18.240 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:25:18.242 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:25:18.242 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:25:18.242 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:25:18.243 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:25:18.243 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:25:18.243 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:25:18.243 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:25:18.243 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:25:18.243 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:25:18.243 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:25:18.243 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:25:18.243 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:25:18.243 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:25:18.243 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:25:18.243 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:25:18.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:25:18.244 INFO:tasks.workunit.client.0.trial112.stderr:start osd.0 2026-02-13T02:25:18.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:25:18.244 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T02:25:18.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:25:18.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:25:18.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:25:18.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:25:18.260 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:18.247+0000 7f2fcf6dea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:18.261 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:18.251+0000 7f2fcf6dea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:18.262 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:18.251+0000 7f2fcf6dea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:18.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:25:18.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:25:18.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:25:18.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:25:18.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:25:18.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:25:18.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:25:18.409 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:25:18.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:25:18.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:25:18.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:25:18.650 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:18.639+0000 7f2fcf6dea40 -1 Falling back to public interface 2026-02-13T02:25:19.277 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:19.263+0000 7f2fcf6dea40 -1 osd.0 19 log_to_monitors true 2026-02-13T02:25:19.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:25:19.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:25:19.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:25:19.567 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:25:19.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:25:19.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:25:19.729 INFO:tasks.workunit.client.0.trial112.stderr:osd.0 up in weight 1 up_from 23 up_thru 23 down_at 20 last_clean_interval [5,19) [v2:127.0.0.1:6800/2056678200,v1:127.0.0.1:6801/2056678200] [v2:127.0.0.1:6802/2056678200,v1:127.0.0.1:6803/2056678200] exists,up 227f61b4-b86c-4e96-8ec7-ee44ffa2dea3 2026-02-13T02:25:19.729 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:25:19.729 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:25:19.729 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:25:19.729 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:25:19.729 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:25:19.729 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:25:19.729 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:25:19.730 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:25:19.731 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:25:19.731 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:25:19.731 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:25:19.731 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:25:19.731 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:25:19.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:25:19.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:25:19.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:25:19.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:25:19.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:25:19.851 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:25:20.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:25:20.006 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:25:20.006 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:25:20.006 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:25:20.006 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:25:20.082 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247810 2026-02-13T02:25:20.082 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247810 2026-02-13T02:25:20.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247810' 2026-02-13T02:25:20.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:25:20.083 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:25:20.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705681 2026-02-13T02:25:20.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705681 2026-02-13T02:25:20.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247810 1-38654705681' 2026-02-13T02:25:20.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:25:20.158 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-98784247810 2026-02-13T02:25:20.159 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:25:20.160 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:25:20.161 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-98784247810 2026-02-13T02:25:20.161 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:25:20.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247810 2026-02-13T02:25:20.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 98784247810' 2026-02-13T02:25:20.163 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 98784247810 2026-02-13T02:25:20.163 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:25:20.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836497 -lt 98784247810 2026-02-13T02:25:20.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:25:21.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:25:21.330 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:25:21.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247811 -lt 98784247810 2026-02-13T02:25:21.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:25:21.496 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705681 2026-02-13T02:25:21.496 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:25:21.497 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:25:21.498 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705681 2026-02-13T02:25:21.499 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:25:21.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705681 2026-02-13T02:25:21.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705681' 2026-02-13T02:25:21.501 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 38654705681 2026-02-13T02:25:21.501 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:25:21.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705682 -lt 38654705681 2026-02-13T02:25:21.674 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:25:21.675 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:25:21.675 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:25:21.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-13T02:25:21.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:25:21.876 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:25:21.876 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:25:21.876 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:25:21.876 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:25:21.876 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:25:21.876 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:25:22.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-13T02:25:22.037 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:25:22.037 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:25:22.038 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:25:22.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-13T02:25:22.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:25:22.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:25:22.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:240: scrub_and_not_schedule: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2026-02-13T02:25:22.258 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:241: scrub_and_not_schedule: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-13T02:25:22.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-13T02:25:22.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:25:22.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:25:22.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:25:22.260 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:25:22.260 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:25:22.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:25:22.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:25:22.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:25:22.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:25:22.372 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:25:22.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:25:22.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:25:22.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:25:22.376 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:25:22.376 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:25:22.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:25:22.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:25:22.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:25:22.378 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:25:22.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:25:22.381 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:25:22.381 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:25:22.387 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:25:22.387 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:25:22.387 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:25:22.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:25:22.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:25:22.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:25:22.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-13T02:25:22.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-13T02:25:22.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-13T02:25:22.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-13T02:25:22.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:25:22.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:25:22.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:25:22.390 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:25:22.390 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:25:22.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:25:22.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:25:22.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:25:22.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:25:22.393 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:25:22.394 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:25:22.395 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:25:22.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:25:22.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:25:22.397 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:25:22.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:25:22.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:25:22.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:25:22.399 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:25:22.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:25:22.401 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:25:22.401 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:25:22.403 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:25:22.403 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:25:22.403 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:25:22.403 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:25:22.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:25:22.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:25:22.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-13T02:25:22.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T02:25:22.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:25:22.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:25:22.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.35207 2026-02-13T02:25:22.409 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T02:25:22.410 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-13T02:25:22.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T02:25:22.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-13T02:25:22.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_unfound_erasure_coded_appends td/osd-scrub-repair 2026-02-13T02:25:22.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:962: TEST_unfound_erasure_coded_appends: unfound_erasure_coded td/osd-scrub-repair false 2026-02-13T02:25:22.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:911: unfound_erasure_coded: local dir=td/osd-scrub-repair 2026-02-13T02:25:22.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:912: unfound_erasure_coded: local allow_overwrites=false 2026-02-13T02:25:22.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:913: unfound_erasure_coded: local poolname=ecpool 2026-02-13T02:25:22.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:914: unfound_erasure_coded: local payload=ABCDEF 2026-02-13T02:25:22.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:916: unfound_erasure_coded: run_mon td/osd-scrub-repair a 2026-02-13T02:25:22.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-13T02:25:22.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-13T02:25:22.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-13T02:25:22.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-13T02:25:22.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-13T02:25:22.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-13T02:25:22.431 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-13T02:25:22.431 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:25:22.431 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:25:22.432 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:25:22.432 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:25:22.432 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:25:22.432 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:25:22.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-13T02:25:22.463 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-13T02:25:22.464 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-13T02:25:22.464 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:25:22.464 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:25:22.464 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-13T02:25:22.465 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-13T02:25:22.465 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:25:22.465 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:25:22.465 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:25:22.466 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:25:22.466 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:25:22.466 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:25:22.466 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:25:22.467 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:25:22.467 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get fsid 2026-02-13T02:25:22.522 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-13T02:25:22.522 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:25:22.522 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:25:22.522 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-13T02:25:22.523 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-13T02:25:22.523 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:25:22.523 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:25:22.523 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:25:22.523 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:25:22.523 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:25:22.523 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:25:22.524 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:25:22.524 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:25:22.524 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get mon_host 2026-02-13T02:25:22.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:917: unfound_erasure_coded: run_mgr td/osd-scrub-repair x 2026-02-13T02:25:22.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-13T02:25:22.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-13T02:25:22.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-13T02:25:22.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-13T02:25:22.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-13T02:25:22.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-13T02:25:22.672 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-13T02:25:22.672 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:25:22.672 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:25:22.673 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:25:22.673 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:25:22.673 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:25:22.673 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:25:22.674 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:25:22.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:25:22.695 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: unfound_erasure_coded: seq 0 3 2026-02-13T02:25:22.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: unfound_erasure_coded: for id in $(seq 0 3) 2026-02-13T02:25:22.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:919: unfound_erasure_coded: run_osd td/osd-scrub-repair 0 2026-02-13T02:25:22.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:25:22.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:25:22.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-13T02:25:22.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:25:22.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:25:22.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:25:22.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:25:22.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:25:22.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:25:22.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:25:22.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:25:22.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:25:22.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:25:22.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:25:22.699 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:25:22.699 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:25:22.699 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:25:22.699 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:25:22.699 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:25:22.699 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:25:22.699 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:25:22.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:25:22.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:25:22.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:25:22.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:25:22.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:25:22.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:25:22.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:25:22.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:25:22.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:25:22.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:25:22.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:25:22.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:25:22.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:25:22.702 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:25:22.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=295373e7-abbb-4538-b02c-37ce559b3542 2026-02-13T02:25:22.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 295373e7-abbb-4538-b02c-37ce559b3542' 2026-02-13T02:25:22.703 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 295373e7-abbb-4538-b02c-37ce559b3542 2026-02-13T02:25:22.703 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:25:22.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCSi45pAN4jKhAAO8Y8+FIGN/Bo/iPorAwtAg== 2026-02-13T02:25:22.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCSi45pAN4jKhAAO8Y8+FIGN/Bo/iPorAwtAg=="}' 2026-02-13T02:25:22.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 295373e7-abbb-4538-b02c-37ce559b3542 -i td/osd-scrub-repair/0/new.json 2026-02-13T02:25:22.808 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:25:22.816 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-13T02:25:22.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCSi45pAN4jKhAAO8Y8+FIGN/Bo/iPorAwtAg== --osd-uuid 295373e7-abbb-4538-b02c-37ce559b3542 2026-02-13T02:25:22.834 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:22.823+0000 7fd7d7a36a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:22.835 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:22.823+0000 7fd7d7a36a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:22.836 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:22.823+0000 7fd7d7a36a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:22.837 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:22.827+0000 7fd7d7a36a40 -1 bdev(0x55952c930c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-13T02:25:22.837 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:22.827+0000 7fd7d7a36a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-13T02:25:23.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-13T02:25:23.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:25:23.501 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T02:25:23.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-13T02:25:23.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:25:23.601 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-13T02:25:23.602 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T02:25:23.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:25:23.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:25:23.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:25:23.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:25:23.618 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:23.607+0000 7fcfabbc6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:23.620 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:23.607+0000 7fcfabbc6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:23.621 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:23.611+0000 7fcfabbc6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:23.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-13T02:25:23.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:25:23.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:25:23.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:25:23.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:25:23.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:25:23.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:25:23.700 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:25:23.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:25:23.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:25:23.795 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:25:24.006 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:23.995+0000 7fcfabbc6a40 -1 Falling back to public interface 2026-02-13T02:25:24.628 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:24.615+0000 7fcfabbc6a40 -1 osd.0 0 log_to_monitors true 2026-02-13T02:25:24.797 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:25:24.797 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:25:24.797 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:25:24.798 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:25:24.799 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:25:24.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:25:24.926 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:25:25.927 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:25:25.928 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:25:25.928 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:25:25.929 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:25:25.929 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:25:25.929 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:25:26.086 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/628628958,v1:127.0.0.1:6801/628628958] [v2:127.0.0.1:6802/628628958,v1:127.0.0.1:6803/628628958] exists,up 295373e7-abbb-4538-b02c-37ce559b3542 2026-02-13T02:25:26.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:25:26.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:25:26.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:25:26.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: unfound_erasure_coded: for id in $(seq 0 3) 2026-02-13T02:25:26.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:919: unfound_erasure_coded: run_osd td/osd-scrub-repair 1 2026-02-13T02:25:26.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:25:26.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:25:26.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-13T02:25:26.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:25:26.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:25:26.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:25:26.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:25:26.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:25:26.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:25:26.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:25:26.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:25:26.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:25:26.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:25:26.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:25:26.092 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:25:26.092 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:25:26.092 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:25:26.092 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:25:26.092 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:25:26.092 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:25:26.092 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:25:26.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:25:26.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:25:26.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:25:26.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:25:26.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:25:26.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:25:26.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:25:26.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:25:26.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:25:26.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:25:26.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:25:26.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:25:26.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:25:26.094 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:25:26.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4a481782-f526-4dce-bb5d-c7e7d2f93143 2026-02-13T02:25:26.094 INFO:tasks.workunit.client.0.trial112.stdout:add osd1 4a481782-f526-4dce-bb5d-c7e7d2f93143 2026-02-13T02:25:26.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 4a481782-f526-4dce-bb5d-c7e7d2f93143' 2026-02-13T02:25:26.095 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:25:26.104 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCWi45pJrHNBRAA44tbMI8yyNyEhnzoQzHtxA== 2026-02-13T02:25:26.105 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCWi45pJrHNBRAA44tbMI8yyNyEhnzoQzHtxA=="}' 2026-02-13T02:25:26.105 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4a481782-f526-4dce-bb5d-c7e7d2f93143 -i td/osd-scrub-repair/1/new.json 2026-02-13T02:25:26.268 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:25:26.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-13T02:25:26.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCWi45pJrHNBRAA44tbMI8yyNyEhnzoQzHtxA== --osd-uuid 4a481782-f526-4dce-bb5d-c7e7d2f93143 2026-02-13T02:25:26.304 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:26.291+0000 7f343c6f5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:26.305 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:26.295+0000 7f343c6f5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:26.306 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:26.295+0000 7f343c6f5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:26.307 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:26.295+0000 7f343c6f5a40 -1 bdev(0x560d5c9fec00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-13T02:25:26.307 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:26.295+0000 7f343c6f5a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-13T02:25:26.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-13T02:25:26.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:25:26.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-13T02:25:26.986 INFO:tasks.workunit.client.0.trial112.stdout:adding osd1 key to auth repository 2026-02-13T02:25:26.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:25:27.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-13T02:25:27.185 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T02:25:27.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:25:27.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:25:27.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:25:27.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:25:27.210 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:27.199+0000 7fdf93fbaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:27.212 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:27.199+0000 7fdf93fbaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:27.213 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:27.203+0000 7fdf93fbaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:27.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-13T02:25:27.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:25:27.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:25:27.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:25:27.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:25:27.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:25:27.352 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:25:27.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:25:27.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:25:27.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:25:27.509 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:25:27.554 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:27.543+0000 7fdf93fbaa40 -1 Falling back to public interface 2026-02-13T02:25:28.195 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:28.183+0000 7fdf93fbaa40 -1 osd.1 0 log_to_monitors true 2026-02-13T02:25:28.511 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:25:28.511 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:25:28.512 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:25:28.512 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:25:28.512 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:25:28.512 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:25:28.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:25:29.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:25:29.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:25:29.669 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:25:29.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:25:29.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:25:29.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:25:29.824 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3606477913,v1:127.0.0.1:6811/3606477913] [v2:127.0.0.1:6812/3606477913,v1:127.0.0.1:6813/3606477913] exists,up 4a481782-f526-4dce-bb5d-c7e7d2f93143 2026-02-13T02:25:29.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:25:29.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:25:29.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:25:29.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: unfound_erasure_coded: for id in $(seq 0 3) 2026-02-13T02:25:29.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:919: unfound_erasure_coded: run_osd td/osd-scrub-repair 2 2026-02-13T02:25:29.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:25:29.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:25:29.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-13T02:25:29.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:25:29.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-13T02:25:29.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:25:29.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:25:29.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:25:29.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:25:29.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-13T02:25:29.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-13T02:25:29.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:25:29.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:25:29.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:25:29.830 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:25:29.830 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:25:29.830 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:25:29.830 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:25:29.830 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:25:29.830 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:25:29.830 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:25:29.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:25:29.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:25:29.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:25:29.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:25:29.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:25:29.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:25:29.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:25:29.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:25:29.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:25:29.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:25:29.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:25:29.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:25:29.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-13T02:25:29.832 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:25:29.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=71c83e33-4023-4ac1-bcd5-cd456cc36d4e 2026-02-13T02:25:29.833 INFO:tasks.workunit.client.0.trial112.stdout:add osd2 71c83e33-4023-4ac1-bcd5-cd456cc36d4e 2026-02-13T02:25:29.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 71c83e33-4023-4ac1-bcd5-cd456cc36d4e' 2026-02-13T02:25:29.833 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:25:29.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCZi45pCN/RMRAAIdkD1OtGxiQGxbKJjiBxyQ== 2026-02-13T02:25:29.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCZi45pCN/RMRAAIdkD1OtGxiQGxbKJjiBxyQ=="}' 2026-02-13T02:25:29.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 71c83e33-4023-4ac1-bcd5-cd456cc36d4e -i td/osd-scrub-repair/2/new.json 2026-02-13T02:25:30.004 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:25:30.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-13T02:25:30.015 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCZi45pCN/RMRAAIdkD1OtGxiQGxbKJjiBxyQ== --osd-uuid 71c83e33-4023-4ac1-bcd5-cd456cc36d4e 2026-02-13T02:25:30.031 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:30.019+0000 7f43ec107a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:30.032 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:30.019+0000 7f43ec107a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:30.033 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:30.023+0000 7f43ec107a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:30.034 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:30.023+0000 7f43ec107a40 -1 bdev(0x55c471908c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-13T02:25:30.034 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:30.023+0000 7f43ec107a40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-13T02:25:30.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-13T02:25:30.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:25:30.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-13T02:25:30.673 INFO:tasks.workunit.client.0.trial112.stdout:adding osd2 key to auth repository 2026-02-13T02:25:30.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:25:30.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-13T02:25:30.881 INFO:tasks.workunit.client.0.trial112.stdout:start osd.2 2026-02-13T02:25:30.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:25:30.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:25:30.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:25:30.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:25:30.896 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:30.883+0000 7f05ce624a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:30.897 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:30.887+0000 7f05ce624a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:30.898 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:30.887+0000 7f05ce624a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:31.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-13T02:25:31.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:25:31.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T02:25:31.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:25:31.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:25:31.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:25:31.046 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:25:31.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:25:31.047 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:25:31.047 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:25:31.202 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:25:31.282 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:31.271+0000 7f05ce624a40 -1 Falling back to public interface 2026-02-13T02:25:31.892 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:31.879+0000 7f05ce624a40 -1 osd.2 0 log_to_monitors true 2026-02-13T02:25:32.203 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:25:32.203 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:25:32.204 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:25:32.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:25:32.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:25:32.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:25:32.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:25:33.369 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:25:33.369 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:25:33.370 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:25:33.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:25:33.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:25:33.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:25:33.534 INFO:tasks.workunit.client.0.trial112.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/996120473,v1:127.0.0.1:6819/996120473] [v2:127.0.0.1:6820/996120473,v1:127.0.0.1:6821/996120473] exists,up 71c83e33-4023-4ac1-bcd5-cd456cc36d4e 2026-02-13T02:25:33.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:25:33.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:25:33.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:25:33.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: unfound_erasure_coded: for id in $(seq 0 3) 2026-02-13T02:25:33.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:919: unfound_erasure_coded: run_osd td/osd-scrub-repair 3 2026-02-13T02:25:33.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:25:33.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:25:33.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-13T02:25:33.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:25:33.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:25:33.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:25:33.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:25:33.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:25:33.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:25:33.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-13T02:25:33.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-13T02:25:33.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:25:33.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:25:33.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:25:33.539 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:25:33.540 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:25:33.540 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:25:33.540 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:25:33.540 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:25:33.540 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:25:33.540 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:25:33.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:25:33.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:25:33.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:25:33.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:25:33.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:25:33.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:25:33.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:25:33.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:25:33.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:25:33.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:25:33.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:25:33.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:25:33.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/3 2026-02-13T02:25:33.542 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:25:33.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=9d536936-ac6e-440a-9e33-b80543efc9b3 2026-02-13T02:25:33.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 9d536936-ac6e-440a-9e33-b80543efc9b3' 2026-02-13T02:25:33.542 INFO:tasks.workunit.client.0.trial112.stdout:add osd3 9d536936-ac6e-440a-9e33-b80543efc9b3 2026-02-13T02:25:33.543 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:25:33.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCdi45p1/Z9IBAA6TyZebgel59OvbWlY9Tnqw== 2026-02-13T02:25:33.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCdi45p1/Z9IBAA6TyZebgel59OvbWlY9Tnqw=="}' 2026-02-13T02:25:33.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 9d536936-ac6e-440a-9e33-b80543efc9b3 -i td/osd-scrub-repair/3/new.json 2026-02-13T02:25:33.717 INFO:tasks.workunit.client.0.trial112.stdout:3 2026-02-13T02:25:33.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/3/new.json 2026-02-13T02:25:33.729 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCdi45p1/Z9IBAA6TyZebgel59OvbWlY9Tnqw== --osd-uuid 9d536936-ac6e-440a-9e33-b80543efc9b3 2026-02-13T02:25:33.752 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:33.739+0000 7fb58f78aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:33.753 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:33.743+0000 7fb58f78aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:33.754 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:33.743+0000 7fb58f78aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:33.755 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:33.743+0000 7fb58f78aa40 -1 bdev(0x55c7cca7cc00 td/osd-scrub-repair/3/block) open stat got: (1) Operation not permitted 2026-02-13T02:25:33.755 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:33.743+0000 7fb58f78aa40 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2026-02-13T02:25:34.382 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2026-02-13T02:25:34.382 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:25:34.383 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-13T02:25:34.384 INFO:tasks.workunit.client.0.trial112.stdout:adding osd3 key to auth repository 2026-02-13T02:25:34.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:25:34.590 INFO:tasks.workunit.client.0.trial112.stdout:start osd.3 2026-02-13T02:25:34.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-13T02:25:34.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:25:34.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:25:34.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:25:34.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:25:34.606 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:34.595+0000 7f1d9f635a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:34.608 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:34.595+0000 7f1d9f635a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:34.609 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:34.599+0000 7f1d9f635a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:34.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-13T02:25:34.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:25:34.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-13T02:25:34.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:25:34.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:25:34.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:25:34.754 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:25:34.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:25:34.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:25:34.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:25:34.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:25:34.978 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:34.967+0000 7f1d9f635a40 -1 Falling back to public interface 2026-02-13T02:25:35.532 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:35.519+0000 7f1d9f635a40 -1 osd.3 0 log_to_monitors true 2026-02-13T02:25:35.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:25:35.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:25:35.920 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:25:35.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:25:35.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:25:35.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:25:36.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:25:37.079 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:25:37.079 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:25:37.079 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:25:37.080 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:25:37.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:25:37.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:25:37.238 INFO:tasks.workunit.client.0.trial112.stdout:osd.3 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2248685503,v1:127.0.0.1:6827/2248685503] [v2:127.0.0.1:6828/2248685503,v1:127.0.0.1:6829/2248685503] exists,up 9d536936-ac6e-440a-9e33-b80543efc9b3 2026-02-13T02:25:37.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:25:37.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:25:37.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:25:37.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: unfound_erasure_coded: create_ec_pool ecpool false k=2 m=2 2026-02-13T02:25:37.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2522: create_ec_pool: local pool_name=ecpool 2026-02-13T02:25:37.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2523: create_ec_pool: shift 2026-02-13T02:25:37.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2524: create_ec_pool: local allow_overwrites=false 2026-02-13T02:25:37.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2525: create_ec_pool: shift 2026-02-13T02:25:37.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2527: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2026-02-13T02:25:37.438 INFO:tasks.workunit.client.0.trial112.stderr:load: isa 2026-02-13T02:25:37.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2529: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2026-02-13T02:25:37.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2026-02-13T02:25:37.742 INFO:tasks.workunit.client.0.trial112.stderr:pool 'ecpool' created 2026-02-13T02:25:37.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:25:38.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2531: create_ec_pool: '[' false = true ']' 2026-02-13T02:25:38.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2535: create_ec_pool: wait_for_clean 2026-02-13T02:25:38.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:25:38.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:25:38.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:25:38.758 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:25:38.759 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:25:38.759 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:25:38.760 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:25:38.760 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:25:38.760 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:25:38.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:25:38.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:25:38.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:25:38.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:25:38.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:25:38.880 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:25:39.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:25:39.036 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:25:39.036 INFO:tasks.workunit.client.0.trial112.stderr:2 2026-02-13T02:25:39.037 INFO:tasks.workunit.client.0.trial112.stderr:3' 2026-02-13T02:25:39.037 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:25:39.037 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:25:39.037 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:25:39.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836484 2026-02-13T02:25:39.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836484 2026-02-13T02:25:39.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484' 2026-02-13T02:25:39.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:25:39.112 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:25:39.184 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705667 2026-02-13T02:25:39.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705667 2026-02-13T02:25:39.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-38654705667' 2026-02-13T02:25:39.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:25:39.185 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:25:39.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574851 2026-02-13T02:25:39.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574851 2026-02-13T02:25:39.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-38654705667 2-55834574851' 2026-02-13T02:25:39.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:25:39.257 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:25:39.331 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=73014444034 2026-02-13T02:25:39.331 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 73014444034 2026-02-13T02:25:39.331 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-38654705667 2-55834574851 3-73014444034' 2026-02-13T02:25:39.331 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:25:39.332 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836484 2026-02-13T02:25:39.332 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:25:39.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:25:39.334 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836484 2026-02-13T02:25:39.334 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:25:39.336 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836484 2026-02-13T02:25:39.336 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-13T02:25:39.336 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836484 2026-02-13T02:25:39.336 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:25:39.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836484 2026-02-13T02:25:39.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:25:40.505 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:25:40.505 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:25:40.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836484 2026-02-13T02:25:40.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:25:41.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:25:41.671 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:25:41.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836484 2026-02-13T02:25:41.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:25:41.839 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705667 2026-02-13T02:25:41.839 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:25:41.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:25:41.841 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705667 2026-02-13T02:25:41.842 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:25:41.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705667 2026-02-13T02:25:41.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705667' 2026-02-13T02:25:41.844 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705667 2026-02-13T02:25:41.844 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:25:42.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705668 -lt 38654705667 2026-02-13T02:25:42.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:25:42.013 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574851 2026-02-13T02:25:42.013 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:25:42.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:25:42.015 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574851 2026-02-13T02:25:42.016 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:25:42.017 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574851 2026-02-13T02:25:42.017 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574851' 2026-02-13T02:25:42.018 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574851 2026-02-13T02:25:42.018 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:25:42.184 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574851 -lt 55834574851 2026-02-13T02:25:42.184 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:25:42.185 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-73014444034 2026-02-13T02:25:42.185 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:25:42.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:25:42.187 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-73014444034 2026-02-13T02:25:42.188 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:25:42.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=73014444034 2026-02-13T02:25:42.190 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.3 seq 73014444034 2026-02-13T02:25:42.190 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 73014444034' 2026-02-13T02:25:42.190 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:25:42.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 73014444034 -lt 73014444034 2026-02-13T02:25:42.350 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:25:42.351 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:25:42.351 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:25:42.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:25:42.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:25:42.540 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:25:42.540 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:25:42.540 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:25:42.540 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:25:42.541 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:25:42.541 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:25:42.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:25:42.695 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:25:42.696 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:25:42.696 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:25:42.893 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:25:42.893 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:25:42.893 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:25:42.893 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2536: create_ec_pool: return 0 2026-02-13T02:25:42.893 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:924: unfound_erasure_coded: add_something td/osd-scrub-repair ecpool 2026-02-13T02:25:42.893 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:25:42.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-13T02:25:42.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2026-02-13T02:25:42.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:25:42.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:25:42.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:25:43.060 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:25:43.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:25:43.261 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:25:43.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:25:43.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:25:43.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-13T02:25:43.293 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: unfound_erasure_coded: get_primary ecpool SOMETHING 2026-02-13T02:25:43.293 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2026-02-13T02:25:43.293 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-13T02:25:43.294 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T02:25:43.294 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2026-02-13T02:25:43.456 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: unfound_erasure_coded: local primary=1 2026-02-13T02:25:43.457 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:927: unfound_erasure_coded: get_osds ecpool SOMETHING 2026-02-13T02:25:43.458 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:927: unfound_erasure_coded: sed -e s/1// 2026-02-13T02:25:43.458 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2026-02-13T02:25:43.458 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-13T02:25:43.458 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2026-02-13T02:25:43.458 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-13T02:25:43.625 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-13T02:25:43.625 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:25:43.625 INFO:tasks.workunit.client.0.trial112.stderr:2 2026-02-13T02:25:43.625 INFO:tasks.workunit.client.0.trial112.stderr:3' 2026-02-13T02:25:43.625 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2 3 2026-02-13T02:25:43.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:927: unfound_erasure_coded: osds=('0' '2' '3') 2026-02-13T02:25:43.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:927: unfound_erasure_coded: local -a osds 2026-02-13T02:25:43.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:928: unfound_erasure_coded: local not_primary_first=0 2026-02-13T02:25:43.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:929: unfound_erasure_coded: local not_primary_second=2 2026-02-13T02:25:43.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:930: unfound_erasure_coded: local not_primary_third=3 2026-02-13T02:25:43.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: unfound_erasure_coded: pids= 2026-02-13T02:25:43.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2026-02-13T02:25:43.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-13T02:25:43.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-13T02:25:43.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 438005"' 2026-02-13T02:25:43.629 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 438005' 2026-02-13T02:25:43.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:937: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2026-02-13T02:25:43.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-13T02:25:43.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-13T02:25:43.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 438006"' 2026-02-13T02:25:43.630 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 438006' 2026-02-13T02:25:43.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:938: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-13T02:25:43.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-13T02:25:43.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-13T02:25:43.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2026-02-13T02:25:43.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/438008: /' 2026-02-13T02:25:43.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 438009"' 2026-02-13T02:25:43.630 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 438009' 2026-02-13T02:25:43.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: unfound_erasure_coded: wait_background pids 2026-02-13T02:25:43.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 438005 438006 438009' 2026-02-13T02:25:43.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-13T02:25:43.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-13T02:25:43.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 438005 2026-02-13T02:25:43.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2026-02-13T02:25:43.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/438011: /' 2026-02-13T02:25:43.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-13T02:25:43.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/438014: /' 2026-02-13T02:25:44.261 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:25:44.261 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:25:44.261 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-13T02:25:44.262 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:25:44.262 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2026-02-13T02:25:44.262 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:25:44.262 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:25:44.262 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-13T02:25:44.262 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:25:44.262 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-13T02:25:44.262 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:25:44.262 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:25:44.262 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:25:44.262 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:25:44.262 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:25:44.263 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:25:44.263 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2026-02-13T02:25:44.263 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:25:44.263 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:25:44.263 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-13T02:25:44.263 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:25:44.263 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:25:44.263 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2026-02-13T02:25:44.263 INFO:tasks.workunit.client.0.trial112.stderr:438014: remove 3#1:eb822e21:::SOMETHING:head# 2026-02-13T02:25:44.263 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-13T02:25:44.263 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:25:44.263 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:25:44.263 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-13T02:25:44.264 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:25:44.264 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:25:44.264 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:25:44.264 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:25:44.265 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:25:44.266 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:25:44.266 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-13T02:25:44.266 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-13T02:25:44.266 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:25:44.266 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:25:44.266 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:25:44.266 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:25:44.266 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:25:44.266 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:25:44.266 INFO:tasks.workunit.client.0.trial112.stderr:438014: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:25:44.266 INFO:tasks.workunit.client.0.trial112.stderr:438014: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:25:44.267 INFO:tasks.workunit.client.0.trial112.stderr:438014: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:25:44.267 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:25:44.267 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:25:44.267 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:25:44.267 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:25:44.267 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:25:44.267 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:25:44.267 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:25:44.267 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:25:44.267 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:25:44.267 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:25:44.267 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:25:44.268 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-13T02:25:44.268 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-13T02:25:44.268 INFO:tasks.workunit.client.0.trial112.stderr:438014: start osd.3 2026-02-13T02:25:44.268 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:25:44.269 INFO:tasks.workunit.client.0.trial112.stderr:4438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:25:44.269 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:25:44.269 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T02:25:44.269 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:25:44.269 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2026-02-13T02:25:44.269 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:25:44.269 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:25:44.270 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T02:25:44.270 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:25:44.270 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T02:25:44.270 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:25:44.270 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:25:44.270 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:25:44.270 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:25:44.270 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:25:44.270 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:25:44.270 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2026-02-13T02:25:44.270 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:25:44.270 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:25:44.271 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:25:44.271 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:25:44.271 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:25:44.271 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2026-02-13T02:25:44.271 INFO:tasks.workunit.client.0.trial112.stderr:438008: remove 1#1:eb822e21:::SOMETHING:head# 2026-02-13T02:25:44.271 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-13T02:25:44.271 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:25:44.271 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:25:44.271 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:25:44.271 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:25:44.271 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:25:44.271 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:25:44.271 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:25:44.273 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:25:44.273 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:25:44.273 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:25:44.273 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:25:44.273 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:25:44.273 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:25:44.273 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:25:44.273 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:25:44.273 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:25:44.274 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:25:44.274 INFO:tasks.workunit.client.0.trial112.stderr:438008: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:25:44.274 INFO:tasks.workunit.client.0.trial112.stderr:438008: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:25:44.274 INFO:tasks.workunit.client.0.trial112.stderr:438008: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:25:44.274 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:25:44.274 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:25:44.274 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:25:44.274 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:25:44.274 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:25:44.274 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:25:44.274 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:25:44.274 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:25:44.274 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:25:44.275 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:25:44.275 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:25:44.275 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:25:44.275 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:25:44.275 INFO:tasks.workunit.client.0.trial112.stderr:438008: start osd.0 2026-02-13T02:25:44.275 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:25:44.293 INFO:tasks.workunit.client.0.trial112.stderr:4438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:25:44.294 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:25:44.294 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-13T02:25:44.294 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:25:44.294 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2026-02-13T02:25:44.294 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:25:44.294 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:25:44.294 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-13T02:25:44.294 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:25:44.294 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2026-02-13T02:25:44.294 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:25:44.294 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:25:44.294 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:25:44.295 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:25:44.295 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:25:44.295 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:25:44.295 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2026-02-13T02:25:44.295 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:25:44.295 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:25:44.295 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-13T02:25:44.295 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:25:44.295 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2026-02-13T02:25:44.295 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2026-02-13T02:25:44.295 INFO:tasks.workunit.client.0.trial112.stderr:438011: remove 2#1:eb822e21:::SOMETHING:head# 2026-02-13T02:25:44.295 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2026-02-13T02:25:44.295 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:25:44.296 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:25:44.296 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-13T02:25:44.296 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:25:44.296 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2026-02-13T02:25:44.296 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:25:44.296 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:25:44.296 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:25:44.297 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:25:44.297 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-13T02:25:44.297 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-13T02:25:44.297 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:25:44.297 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:25:44.297 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:25:44.297 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:25:44.297 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:25:44.297 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:25:44.297 INFO:tasks.workunit.client.0.trial112.stderr:438011: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:25:44.297 INFO:tasks.workunit.client.0.trial112.stderr:438011: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:25:44.298 INFO:tasks.workunit.client.0.trial112.stderr:438011: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:25:44.298 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:25:44.298 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:25:44.298 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:25:44.298 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:25:44.298 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:25:44.298 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:25:44.298 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:25:44.298 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:25:44.298 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:25:44.298 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:25:44.298 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:25:44.298 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2026-02-13T02:25:44.299 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-13T02:25:44.299 INFO:tasks.workunit.client.0.trial112.stderr:438011: start osd.2 2026-02-13T02:25:44.299 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:25:45.752 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-13T02:25:45.753 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-13T02:25:45.753 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:25:45.753 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:25:45.753 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:25:45.753 INFO:tasks.workunit.client.0.trial112.stderr:438014: 2026-02-13T02:25:44.279+0000 7f95c423ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:45.753 INFO:tasks.workunit.client.0.trial112.stderr:438014: 2026-02-13T02:25:44.279+0000 7f95c423ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:45.753 INFO:tasks.workunit.client.0.trial112.stderr:438014: 2026-02-13T02:25:44.279+0000 7f95c423ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:45.753 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-13T02:25:45.753 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:25:45.754 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-13T02:25:45.754 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:25:45.754 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:25:45.754 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:25:45.754 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:25:45.754 INFO:tasks.workunit.client.0.trial112.stderr:438014: 0 2026-02-13T02:25:45.754 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:25:45.754 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:25:45.754 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:25:45.754 INFO:tasks.workunit.client.0.trial112.stderr:438014: 2026-02-13T02:25:44.747+0000 7f95c423ba40 -1 Falling back to public interface 2026-02-13T02:25:45.754 INFO:tasks.workunit.client.0.trial112.stderr:438014: 2026-02-13T02:25:45.419+0000 7f95c423ba40 -1 osd.3 24 log_to_monitors true 2026-02-13T02:25:45.755 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:25:45.755 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:25:45.755 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:25:45.755 INFO:tasks.workunit.client.0.trial112.stderr:438014: 1 2026-02-13T02:25:45.755 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:25:45.755 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:25:45.755 INFO:tasks.workunit.client.0.trial112.stderr:438014: osd.3 up in weight 1 up_from 29 up_thru 0 down_at 25 last_clean_interval [17,24) [v2:127.0.0.1:6818/402061844,v1:127.0.0.1:6819/402061844] [v2:127.0.0.1:6820/402061844,v1:127.0.0.1:6821/402061844] exists,up 9d536936-ac6e-440a-9e33-b80543efc9b3 2026-02-13T02:25:45.755 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:25:45.755 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:25:45.755 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:25:45.755 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:25:45.756 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:25:45.756 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:25:45.756 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:25:45.756 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:25:45.756 INFO:tasks.workunit.client.0.trial112.stderr:438014: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:25:45.764 INFO:tasks.workunit.client.0.trial112.stderr:438014: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_ti38008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T02:25:45.764 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:25:45.764 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:25:45.764 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:25:45.764 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:25:45.764 INFO:tasks.workunit.client.0.trial112.stderr:438008: 2026-02-13T02:25:44.275+0000 7f323c3a6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:45.764 INFO:tasks.workunit.client.0.trial112.stderr:438008: 2026-02-13T02:25:44.279+0000 7f323c3a6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:45.765 INFO:tasks.workunit.client.0.trial112.stderr:438008: 2026-02-13T02:25:44.279+0000 7f323c3a6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:45.765 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:25:45.765 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:25:45.765 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:25:45.765 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:25:45.765 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:25:45.765 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:25:45.765 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:25:45.765 INFO:tasks.workunit.client.0.trial112.stderr:438008: 0 2026-02-13T02:25:45.765 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:25:45.765 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:25:45.766 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:25:45.766 INFO:tasks.workunit.client.0.trial112.stderr:438008: 2026-02-13T02:25:44.699+0000 7f323c3a6a40 -1 Falling back to public interface 2026-02-13T02:25:45.766 INFO:tasks.workunit.client.0.trial112.stderr:438008: 2026-02-13T02:25:45.335+0000 7f323c3a6a40 -1 osd.0 24 log_to_monitors true 2026-02-13T02:25:45.766 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:25:45.766 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:25:45.766 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:25:45.766 INFO:tasks.workunit.client.0.trial112.stderr:438008: 1 2026-02-13T02:25:45.766 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:25:45.766 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:25:45.766 INFO:tasks.workunit.client.0.trial112.stderr:438008: osd.0 up in weight 1 up_from 28 up_thru 0 down_at 25 last_clean_interval [5,24) [v2:127.0.0.1:6800/387124191,v1:127.0.0.1:6801/387124191] [v2:127.0.0.1:6802/387124191,v1:127.0.0.1:6803/387124191] exists,up 295373e7-abbb-4538-b02c-37ce559b3542 2026-02-13T02:25:45.766 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:25:45.767 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:25:45.767 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:25:45.767 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:25:45.767 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:25:45.767 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:25:45.767 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:25:45.767 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:25:45.767 INFO:tasks.workunit.client.0.trial112.stderr:438008: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:25:45.787 INFO:tasks.workunit.client.0.trial112.stderr:438008: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_tim38011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2026-02-13T02:25:45.787 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-13T02:25:45.787 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:25:45.787 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:25:45.787 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:25:45.788 INFO:tasks.workunit.client.0.trial112.stderr:438011: 2026-02-13T02:25:44.299+0000 7f770c118a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:45.788 INFO:tasks.workunit.client.0.trial112.stderr:438011: 2026-02-13T02:25:44.303+0000 7f770c118a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:45.788 INFO:tasks.workunit.client.0.trial112.stderr:438011: 2026-02-13T02:25:44.303+0000 7f770c118a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:45.788 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-13T02:25:45.788 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:25:45.788 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T02:25:45.788 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:25:45.788 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:25:45.788 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:25:45.788 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:25:45.788 INFO:tasks.workunit.client.0.trial112.stderr:438011: 0 2026-02-13T02:25:45.788 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:25:45.789 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:25:45.789 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:25:45.789 INFO:tasks.workunit.client.0.trial112.stderr:438011: 2026-02-13T02:25:44.771+0000 7f770c118a40 -1 Falling back to public interface 2026-02-13T02:25:45.789 INFO:tasks.workunit.client.0.trial112.stderr:438011: 2026-02-13T02:25:45.423+0000 7f770c118a40 -1 osd.2 24 log_to_monitors true 2026-02-13T02:25:45.789 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:25:45.789 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:25:45.789 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:25:45.789 INFO:tasks.workunit.client.0.trial112.stderr:438011: 1 2026-02-13T02:25:45.789 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:25:45.789 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:25:45.789 INFO:tasks.workunit.client.0.trial112.stderr:438011: osd.2 up in weight 1 up_from 29 up_thru 0 down_at 25 last_clean_interval [13,24) [v2:127.0.0.1:6826/1809780862,v1:127.0.0.1:6827/1809780862] [v2:127.0.0.1:6828/1809780862,v1:127.0.0.1:6829/1809780862] exists,up 71c83e33-4023-4ac1-bcd5-cd456cc36d4e 2026-02-13T02:25:45.789 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:25:45.789 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:25:45.790 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:25:45.790 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:25:45.790 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:25:45.790 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:25:45.790 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:25:45.790 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:25:45.790 INFO:tasks.workunit.client.0.trial112.stderr:438011: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:25:46.354 INFO:tasks.workunit.client.0.trial112.stderr:438011: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: gemeout_delays: echo true 2026-02-13T02:25:46.355 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:25:46.355 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:25:46.355 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:25:46.355 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:25:46.355 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:25:46.355 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:25:46.355 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:25:46.355 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:25:46.356 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:25:46.356 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:25:46.356 INFO:tasks.workunit.client.0.trial112.stderr:438014: 1 2026-02-13T02:25:46.356 INFO:tasks.workunit.client.0.trial112.stderr:438014: 2 2026-02-13T02:25:46.356 INFO:tasks.workunit.client.0.trial112.stderr:438014: 3' 2026-02-13T02:25:46.356 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:25:46.356 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:25:46.356 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:25:46.356 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=120259084290 2026-02-13T02:25:46.356 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 120259084290 2026-02-13T02:25:46.357 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-120259084290' 2026-02-13T02:25:46.357 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:25:46.357 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:25:46.357 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705670 2026-02-13T02:25:46.357 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705670 2026-02-13T02:25:46.357 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-120259084290 1-38654705670' 2026-02-13T02:25:46.357 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:25:46.357 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:25:46.357 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=124554051586 2026-02-13T02:25:46.357 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 124554051586 2026-02-13T02:25:46.358 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-120259084290 1-38654705670 2-124554051586' 2026-02-13T02:25:46.358 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:25:46.358 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:25:46.358 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=124554051586 2026-02-13T02:25:46.358 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 124554051586 2026-02-13T02:25:46.358 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-120259084290 1-38654705670 2-124554051586 3-124554051586' 2026-02-13T02:25:46.358 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:25:46.358 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-120259084290 2026-02-13T02:25:46.368 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubueout_delays: echo true 2026-02-13T02:25:46.368 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:25:46.368 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:25:46.368 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:25:46.368 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:25:46.368 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:25:46.368 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:25:46.368 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:25:46.368 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:25:46.369 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:25:46.369 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:25:46.369 INFO:tasks.workunit.client.0.trial112.stderr:438008: 1 2026-02-13T02:25:46.369 INFO:tasks.workunit.client.0.trial112.stderr:438008: 2 2026-02-13T02:25:46.369 INFO:tasks.workunit.client.0.trial112.stderr:438008: 3' 2026-02-13T02:25:46.369 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:25:46.369 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:25:46.369 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:25:46.369 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=120259084291 2026-02-13T02:25:46.369 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 120259084291 2026-02-13T02:25:46.369 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-120259084291' 2026-02-13T02:25:46.370 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:25:46.370 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:25:46.370 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705671 2026-02-13T02:25:46.370 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705671 2026-02-13T02:25:46.370 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-120259084291 1-38654705671' 2026-02-13T02:25:46.370 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:25:46.370 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:25:46.370 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=124554051587 2026-02-13T02:25:46.370 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 124554051587 2026-02-13T02:25:46.370 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-120259084291 1-38654705671 2-124554051587' 2026-02-13T02:25:46.370 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:25:46.371 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:25:46.371 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=124554051587 2026-02-13T02:25:46.371 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 124554051587 2026-02-13T02:25:46.371 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-120259084291 1-38654705671 2-124554051587 3-124554051587' 2026-02-13T02:25:46.371 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:25:46.371 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-120259084291 2026-02-13T02:25:46.372 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubunt_timeout_delays: echo true 2026-02-13T02:25:46.372 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:25:46.372 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:25:46.372 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:25:46.373 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:25:46.373 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:25:46.373 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:25:46.373 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:25:46.373 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:25:46.373 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:25:46.373 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:25:46.373 INFO:tasks.workunit.client.0.trial112.stderr:438011: 1 2026-02-13T02:25:46.373 INFO:tasks.workunit.client.0.trial112.stderr:438011: 2 2026-02-13T02:25:46.373 INFO:tasks.workunit.client.0.trial112.stderr:438011: 3' 2026-02-13T02:25:46.373 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:25:46.374 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:25:46.374 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:25:46.374 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=120259084292 2026-02-13T02:25:46.374 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 120259084292 2026-02-13T02:25:46.374 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-120259084292' 2026-02-13T02:25:46.374 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:25:46.374 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:25:46.374 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705672 2026-02-13T02:25:46.374 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705672 2026-02-13T02:25:46.374 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-120259084292 1-38654705672' 2026-02-13T02:25:46.374 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:25:46.375 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:25:46.375 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=124554051588 2026-02-13T02:25:46.375 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 124554051588 2026-02-13T02:25:46.375 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-120259084292 1-38654705672 2-124554051588' 2026-02-13T02:25:46.375 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:25:46.375 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:25:46.375 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=124554051588 2026-02-13T02:25:46.375 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 124554051588 2026-02-13T02:25:46.375 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-120259084292 1-38654705672 2-124554051588 3-124554051588' 2026-02-13T02:25:46.375 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:25:46.375 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-120259084292 2026-02-13T02:25:48.018 INFO:tasks.workunit.client.0.trial112.stderr:438011: //homentu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:25:48.019 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:25:48.019 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-120259084290 2026-02-13T02:25:48.019 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:25:48.019 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=120259084290 2026-02-13T02:25:48.019 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 120259084290' 2026-02-13T02:25:48.019 INFO:tasks.workunit.client.0.trial112.stderr:438014: waiting osd.0 seq 120259084290 2026-02-13T02:25:48.019 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:25:48.019 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 120259084290 2026-02-13T02:25:48.019 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:25:48.019 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:25:48.019 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:25:48.020 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 120259084292 -lt 120259084290 2026-02-13T02:25:48.020 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:25:48.020 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705670 2026-02-13T02:25:48.020 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:25:48.020 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:25:48.020 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705670 2026-02-13T02:25:48.020 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:25:48.020 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705670 2026-02-13T02:25:48.020 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705670' 2026-02-13T02:25:48.020 INFO:tasks.workunit.client.0.trial112.stderr:438014: waiting osd.1 seq 38654705670 2026-02-13T02:25:48.020 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:25:48.020 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705672 -lt 38654705670 2026-02-13T02:25:48.020 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:25:48.021 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-124554051586 2026-02-13T02:25:48.021 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:25:48.021 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:25:48.021 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-124554051586 2026-02-13T02:25:48.021 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:25:48.021 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=124554051586 2026-02-13T02:25:48.021 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 124554051586' 2026-02-13T02:25:48.021 INFO:tasks.workunit.client.0.trial112.stderr:438014: waiting osd.2 seq 124554051586 2026-02-13T02:25:48.021 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:25:48.021 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 124554051588 -lt 124554051586 2026-02-13T02:25:48.021 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:25:48.037 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:22tu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:25:48.037 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:25:48.038 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-120259084291 2026-02-13T02:25:48.038 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:25:48.038 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=120259084291 2026-02-13T02:25:48.038 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 120259084291' 2026-02-13T02:25:48.038 INFO:tasks.workunit.client.0.trial112.stderr:438008: waiting osd.0 seq 120259084291 2026-02-13T02:25:48.038 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:25:48.038 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 120259084291 2026-02-13T02:25:48.038 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:25:48.038 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:25:48.038 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:25:48.038 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 120259084292 -lt 120259084291 2026-02-13T02:25:48.039 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:25:48.039 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705671 2026-02-13T02:25:48.039 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:25:48.039 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:25:48.039 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705671 2026-02-13T02:25:48.039 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:25:48.039 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705671 2026-02-13T02:25:48.039 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705671' 2026-02-13T02:25:48.039 INFO:tasks.workunit.client.0.trial112.stderr:438008: waiting osd.1 seq 38654705671 2026-02-13T02:25:48.039 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:25:48.039 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705672 -lt 38654705671 2026-02-13T02:25:48.040 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:25:48.040 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-124554051587 2026-02-13T02:25:48.040 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:25:48.040 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:25:48.040 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-124554051587 2026-02-13T02:25:48.040 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:25:48.040 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=124554051587 2026-02-13T02:25:48.040 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 124554051587' 2026-02-13T02:25:48.040 INFO:tasks.workunit.client.0.trial112.stderr:438008: waiting osd.2 seq 124554051587 2026-02-13T02:25:48.040 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:25:48.040 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 124554051588 -lt 124554051587 2026-02-13T02:25:48.041 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:25:48.043 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:226/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:25:48.043 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:25:48.043 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-120259084292 2026-02-13T02:25:48.043 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:25:48.043 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=120259084292 2026-02-13T02:25:48.043 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 120259084292' 2026-02-13T02:25:48.043 INFO:tasks.workunit.client.0.trial112.stderr:438011: waiting osd.0 seq 120259084292 2026-02-13T02:25:48.044 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:25:48.044 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 120259084292 2026-02-13T02:25:48.044 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:25:48.044 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:25:48.044 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:25:48.044 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 120259084292 -lt 120259084292 2026-02-13T02:25:48.044 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:25:48.044 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705672 2026-02-13T02:25:48.044 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:25:48.044 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:25:48.044 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705672 2026-02-13T02:25:48.045 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:25:48.045 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705672 2026-02-13T02:25:48.045 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705672' 2026-02-13T02:25:48.045 INFO:tasks.workunit.client.0.trial112.stderr:438011: waiting osd.1 seq 38654705672 2026-02-13T02:25:48.045 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:25:48.045 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705672 -lt 38654705672 2026-02-13T02:25:48.045 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:25:48.045 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-124554051588 2026-02-13T02:25:48.045 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:25:48.045 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:25:48.045 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-124554051588 2026-02-13T02:25:48.046 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:25:48.046 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=124554051588 2026-02-13T02:25:48.046 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 124554051588' 2026-02-13T02:25:48.046 INFO:tasks.workunit.client.0.trial112.stderr:438011: waiting osd.2 seq 124554051588 2026-02-13T02:25:48.046 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:25:48.046 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 124554051588 -lt 124554051588 2026-02-13T02:25:48.046 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:25:48.733 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.s67: flush_pg_stats: echo 3-124554051586 2026-02-13T02:25:48.734 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:25:48.734 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:25:48.734 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-124554051586 2026-02-13T02:25:48.734 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:25:48.734 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=124554051586 2026-02-13T02:25:48.734 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 124554051586' 2026-02-13T02:25:48.734 INFO:tasks.workunit.client.0.trial112.stderr:438014: waiting osd.3 seq 124554051586 2026-02-13T02:25:48.734 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:25:48.734 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 124554051588 -lt 124554051586 2026-02-13T02:25:48.734 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:25:48.735 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:25:48.735 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:25:48.735 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:25:48.735 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:25:48.735 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:25:48.735 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:25:48.735 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:25:48.735 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:25:48.735 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:25:48.735 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:25:48.736 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:25:48.736 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:25:48.736 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:25:48.736 INFO:tasks.workunit.client.0.trial112.stderr:438014: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:25:48.736 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:25:48.736 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:25:48.736 INFO:tasks.workunit.client.0.trial112.stderr:438014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:25:48.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-13T02:25:48.765 INFO:tasks.workunit.client.0.trial112.stderr:7: flush_pg_stats: echo 3-124554051587 2026-02-13T02:25:48.765 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:25:48.765 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:25:48.765 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-124554051587 2026-02-13T02:25:48.765 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:25:48.765 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=124554051587 2026-02-13T02:25:48.765 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 124554051587' 2026-02-13T02:25:48.765 INFO:tasks.workunit.client.0.trial112.stderr:438008: waiting osd.3 seq 124554051587 2026-02-13T02:25:48.765 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:25:48.765 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 124554051588 -lt 124554051587 2026-02-13T02:25:48.766 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:25:48.766 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:25:48.766 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:25:48.766 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:25:48.766 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:25:48.766 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:25:48.766 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:25:48.766 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:25:48.766 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:25:48.766 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:25:48.766 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:25:48.766 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:25:48.766 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:25:48.767 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:25:48.767 INFO:tasks.workunit.client.0.trial112.stderr:438008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:25:48.767 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:25:48.767 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:25:48.767 INFO:tasks.workunit.client.0.trial112.stderr:438008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:25:48.767 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-13T02:25:48.768 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-13T02:25:48.768 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 438006 2026-02-13T02:25:48.780 INFO:tasks.workunit.client.0.trial112.stderr:h:2267: flush_pg_stats: echo 3-124554051588 2026-02-13T02:25:48.781 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:25:48.781 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:25:48.781 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-124554051588 2026-02-13T02:25:48.781 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:25:48.781 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=124554051588 2026-02-13T02:25:48.781 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 124554051588' 2026-02-13T02:25:48.781 INFO:tasks.workunit.client.0.trial112.stderr:438011: waiting osd.3 seq 124554051588 2026-02-13T02:25:48.781 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:25:48.781 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 124554051588 -lt 124554051588 2026-02-13T02:25:48.781 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:25:48.781 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:25:48.781 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:25:48.782 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:25:48.782 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:25:48.782 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:25:48.782 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:25:48.782 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:25:48.782 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:25:48.782 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:25:48.782 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:25:48.782 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:25:48.782 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:25:48.782 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:25:48.782 INFO:tasks.workunit.client.0.trial112.stderr:438011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:25:48.783 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:25:48.783 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:25:48.783 INFO:tasks.workunit.client.0.trial112.stderr:438011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:25:48.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-13T02:25:48.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-13T02:25:48.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 438009 2026-02-13T02:25:48.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-13T02:25:48.783 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-13T02:25:48.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-13T02:25:48.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: unfound_erasure_coded: return_code=0 2026-02-13T02:25:48.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:941: unfound_erasure_coded: '[' 0 -ne 0 ']' 2026-02-13T02:25:48.783 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: unfound_erasure_coded: get_pg ecpool SOMETHING 2026-02-13T02:25:48.783 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-13T02:25:48.784 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-13T02:25:48.784 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-13T02:25:48.784 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T02:25:48.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: unfound_erasure_coded: local pg=1.0 2026-02-13T02:25:48.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:947: unfound_erasure_coded: repair 1.0 2026-02-13T02:25:48.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=1.0 2026-02-13T02:25:48.946 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 1.0 2026-02-13T02:25:48.947 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:25:48.947 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:25:48.947 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:25:48.947 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:25:49.108 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-13T02:25:37.735287+0000 2026-02-13T02:25:49.108 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 1.0 2026-02-13T02:25:49.257 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 1.0s0 on osd.1 to repair 2026-02-13T02:25:49.267 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 1.0 2026-02-13T02:25:37.735287+0000 2026-02-13T02:25:49.267 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-13T02:25:49.267 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:25:37.735287+0000 2026-02-13T02:25:49.267 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:25:49.267 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:25:49.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:25:49.268 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:25:49.268 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:25:49.268 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:25:49.268 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:25:49.269 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:25:49.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:25:37.735287+0000 '>' 2026-02-13T02:25:37.735287+0000 2026-02-13T02:25:49.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:25:50.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:25:50.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:25:50.432 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:25:50.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:25:50.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:25:50.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:25:50.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:25:50.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:25:37.735287+0000 '>' 2026-02-13T02:25:37.735287+0000 2026-02-13T02:25:50.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:25:51.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:25:51.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:25:51.590 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:25:51.590 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:25:51.590 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:25:51.591 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:25:51.591 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:25:51.748 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:25:37.735287+0000 '>' 2026-02-13T02:25:37.735287+0000 2026-02-13T02:25:51.748 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:25:52.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:25:52.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:25:52.750 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:25:52.750 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:25:52.751 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:25:52.751 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:25:52.751 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:25:52.906 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:25:37.735287+0000 '>' 2026-02-13T02:25:37.735287+0000 2026-02-13T02:25:52.906 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:25:53.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:25:53.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:25:53.908 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:25:53.908 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:25:53.908 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:25:53.909 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:25:53.909 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:25:54.064 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:25:37.735287+0000 '>' 2026-02-13T02:25:37.735287+0000 2026-02-13T02:25:54.064 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:25:55.065 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:25:55.065 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:25:55.066 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:25:55.066 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:25:55.066 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:25:55.066 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:25:55.066 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:25:55.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:25:50.070649+0000 '>' 2026-02-13T02:25:37.735287+0000 2026-02-13T02:25:55.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:25:55.227 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:952: unfound_erasure_coded: seq 1 60 2026-02-13T02:25:55.228 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:952: unfound_erasure_coded: for f in `seq 1 60` 2026-02-13T02:25:55.228 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: unfound_erasure_coded: ceph -s 2026-02-13T02:25:55.228 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: unfound_erasure_coded: grep '1/1 objects unfound' 2026-02-13T02:25:55.417 INFO:tasks.workunit.client.0.trial112.stdout: 1/1 objects unfound (100.000%) 2026-02-13T02:25:55.417 INFO:tasks.workunit.client.0.trial112.stdout: 1/1 objects unfound (100.000%) 2026-02-13T02:25:55.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: unfound_erasure_coded: break 2026-02-13T02:25:55.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:956: unfound_erasure_coded: ceph -s 2026-02-13T02:25:55.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:956: unfound_erasure_coded: grep '4 up' 2026-02-13T02:25:55.607 INFO:tasks.workunit.client.0.trial112.stdout: osd: 4 osds: 4 up (since 9s), 4 in (since 21s) 2026-02-13T02:25:55.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: unfound_erasure_coded: ceph -s 2026-02-13T02:25:55.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: unfound_erasure_coded: grep '4 in' 2026-02-13T02:25:55.799 INFO:tasks.workunit.client.0.trial112.stdout: osd: 4 osds: 4 up (since 10s), 4 in (since 22s) 2026-02-13T02:25:55.799 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: unfound_erasure_coded: ceph -s 2026-02-13T02:25:55.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: unfound_erasure_coded: grep '1/1 objects unfound' 2026-02-13T02:25:55.987 INFO:tasks.workunit.client.0.trial112.stdout: 1/1 objects unfound (100.000%) 2026-02-13T02:25:55.988 INFO:tasks.workunit.client.0.trial112.stdout: 1/1 objects unfound (100.000%) 2026-02-13T02:25:55.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-13T02:25:55.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:25:55.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:25:55.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:25:55.989 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:25:55.989 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:25:55.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:25:55.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:25:55.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:25:56.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:25:56.102 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:25:56.104 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:25:56.104 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:25:56.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:25:56.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:25:56.106 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:25:56.108 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:25:56.108 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:25:56.108 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:25:56.109 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:25:56.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:25:56.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:25:56.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:25:56.124 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:25:56.124 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:25:56.124 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:25:56.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:25:56.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:25:56.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:25:56.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-13T02:25:56.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-13T02:25:56.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-13T02:25:56.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-13T02:25:56.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:25:56.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:25:56.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:25:56.127 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:25:56.128 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:25:56.128 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:25:56.128 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:25:56.128 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:25:56.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:25:56.130 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:25:56.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:25:56.132 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:25:56.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:25:56.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:25:56.134 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:25:56.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:25:56.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:25:56.136 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:25:56.136 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:25:56.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:25:56.138 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:25:56.138 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:25:56.140 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:25:56.140 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:25:56.140 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:25:56.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:25:56.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:25:56.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:25:56.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-13T02:25:56.143 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T02:25:56.143 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:25:56.143 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:25:56.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.35207 2026-02-13T02:25:56.145 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T02:25:56.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-13T02:25:56.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T02:25:56.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-13T02:25:56.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_unfound_erasure_coded_overwrites td/osd-scrub-repair 2026-02-13T02:25:56.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:966: TEST_unfound_erasure_coded_overwrites: '[' true = true ']' 2026-02-13T02:25:56.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:967: TEST_unfound_erasure_coded_overwrites: unfound_erasure_coded td/osd-scrub-repair true 2026-02-13T02:25:56.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:911: unfound_erasure_coded: local dir=td/osd-scrub-repair 2026-02-13T02:25:56.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:912: unfound_erasure_coded: local allow_overwrites=true 2026-02-13T02:25:56.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:913: unfound_erasure_coded: local poolname=ecpool 2026-02-13T02:25:56.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:914: unfound_erasure_coded: local payload=ABCDEF 2026-02-13T02:25:56.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:916: unfound_erasure_coded: run_mon td/osd-scrub-repair a 2026-02-13T02:25:56.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-13T02:25:56.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-13T02:25:56.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-13T02:25:56.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-13T02:25:56.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-13T02:25:56.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-13T02:25:56.168 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-13T02:25:56.168 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:25:56.168 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:25:56.168 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:25:56.168 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:25:56.168 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:25:56.169 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:25:56.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-13T02:25:56.196 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-13T02:25:56.197 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-13T02:25:56.198 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:25:56.198 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:25:56.198 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-13T02:25:56.198 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-13T02:25:56.199 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:25:56.199 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:25:56.199 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:25:56.199 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:25:56.200 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:25:56.200 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:25:56.200 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:25:56.200 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:25:56.200 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get fsid 2026-02-13T02:25:56.255 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-13T02:25:56.255 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:25:56.256 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:25:56.256 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-13T02:25:56.256 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-13T02:25:56.257 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:25:56.257 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:25:56.257 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:25:56.257 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:25:56.257 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:25:56.257 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:25:56.257 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35207/ceph-mon.a.asok 2026-02-13T02:25:56.258 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:25:56.258 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35207/ceph-mon.a.asok config get mon_host 2026-02-13T02:25:56.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:917: unfound_erasure_coded: run_mgr td/osd-scrub-repair x 2026-02-13T02:25:56.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-13T02:25:56.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-13T02:25:56.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-13T02:25:56.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-13T02:25:56.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-13T02:25:56.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-13T02:25:56.405 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-13T02:25:56.405 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:25:56.405 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:25:56.405 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:25:56.406 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:25:56.406 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:25:56.406 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:25:56.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:25:56.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:25:56.427 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: unfound_erasure_coded: seq 0 3 2026-02-13T02:25:56.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: unfound_erasure_coded: for id in $(seq 0 3) 2026-02-13T02:25:56.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:919: unfound_erasure_coded: run_osd td/osd-scrub-repair 0 2026-02-13T02:25:56.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:25:56.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:25:56.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-13T02:25:56.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:25:56.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:25:56.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:25:56.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:25:56.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:25:56.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:25:56.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:25:56.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:25:56.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:25:56.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:25:56.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:25:56.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:25:56.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:25:56.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:25:56.433 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:25:56.433 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:25:56.433 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:25:56.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:25:56.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:25:56.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:25:56.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:25:56.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:25:56.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:25:56.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:25:56.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:25:56.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:25:56.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:25:56.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:25:56.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:25:56.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:25:56.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:25:56.435 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:25:56.435 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=339d5a6c-c266-4659-9a1d-9fee970374d4 2026-02-13T02:25:56.435 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 339d5a6c-c266-4659-9a1d-9fee970374d4' 2026-02-13T02:25:56.436 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 339d5a6c-c266-4659-9a1d-9fee970374d4 2026-02-13T02:25:56.436 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:25:56.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC0i45pQEEsGhAAtJFHUsdVNozTN8v7fB9Mzw== 2026-02-13T02:25:56.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC0i45pQEEsGhAAtJFHUsdVNozTN8v7fB9Mzw=="}' 2026-02-13T02:25:56.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 339d5a6c-c266-4659-9a1d-9fee970374d4 -i td/osd-scrub-repair/0/new.json 2026-02-13T02:25:56.539 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:25:56.547 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-13T02:25:56.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC0i45pQEEsGhAAtJFHUsdVNozTN8v7fB9Mzw== --osd-uuid 339d5a6c-c266-4659-9a1d-9fee970374d4 2026-02-13T02:25:56.564 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:56.551+0000 7fe0f5b68a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:56.565 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:56.555+0000 7fe0f5b68a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:56.566 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:56.555+0000 7fe0f5b68a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:56.567 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:56.555+0000 7fe0f5b68a40 -1 bdev(0x562d2cb78c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-13T02:25:56.567 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:56.555+0000 7fe0f5b68a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-13T02:25:57.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-13T02:25:57.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:25:57.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-13T02:25:57.354 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T02:25:57.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:25:57.457 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T02:25:57.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-13T02:25:57.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:25:57.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:25:57.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:25:57.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:25:57.481 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:57.471+0000 7f02d9c80a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:57.483 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:57.471+0000 7f02d9c80a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:57.485 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:57.475+0000 7f02d9c80a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:25:57.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-13T02:25:57.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:25:57.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:25:57.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:25:57.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:25:57.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:25:57.555 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:25:57.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:25:57.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:25:57.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:25:57.644 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:25:57.954 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:57.943+0000 7f02d9c80a40 -1 Falling back to public interface 2026-02-13T02:25:58.646 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:25:58.646 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:25:58.646 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:25:58.647 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:25:58.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:25:58.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:25:58.735 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:25:58.950 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:25:58.939+0000 7f02d9c80a40 -1 osd.0 0 log_to_monitors true 2026-02-13T02:25:59.737 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:25:59.737 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:25:59.737 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:25:59.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:25:59.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:25:59.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:25:59.899 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3390270844,v1:127.0.0.1:6801/3390270844] [v2:127.0.0.1:6802/3390270844,v1:127.0.0.1:6803/3390270844] exists,up 339d5a6c-c266-4659-9a1d-9fee970374d4 2026-02-13T02:25:59.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:25:59.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:25:59.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:25:59.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: unfound_erasure_coded: for id in $(seq 0 3) 2026-02-13T02:25:59.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:919: unfound_erasure_coded: run_osd td/osd-scrub-repair 1 2026-02-13T02:25:59.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:25:59.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:25:59.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-13T02:25:59.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:25:59.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-13T02:25:59.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:25:59.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:25:59.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:25:59.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:25:59.904 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-13T02:25:59.904 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-13T02:25:59.904 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:25:59.904 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:25:59.904 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:25:59.904 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:25:59.904 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:25:59.904 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:25:59.904 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:25:59.904 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:25:59.904 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:25:59.904 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:25:59.904 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:25:59.905 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:25:59.905 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:25:59.905 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:25:59.905 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:25:59.905 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:25:59.905 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:25:59.905 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:25:59.905 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:25:59.905 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:25:59.905 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:25:59.905 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:25:59.905 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-13T02:25:59.906 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:25:59.906 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7067c579-51ca-42f1-b536-5bd2c1ab0930 2026-02-13T02:25:59.906 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 7067c579-51ca-42f1-b536-5bd2c1ab0930' 2026-02-13T02:25:59.907 INFO:tasks.workunit.client.0.trial112.stdout:add osd1 7067c579-51ca-42f1-b536-5bd2c1ab0930 2026-02-13T02:25:59.907 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:25:59.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC3i45pa1k8NhAA9l6zTOSbqGVDCHbO4D3sPA== 2026-02-13T02:25:59.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC3i45pa1k8NhAA9l6zTOSbqGVDCHbO4D3sPA=="}' 2026-02-13T02:25:59.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7067c579-51ca-42f1-b536-5bd2c1ab0930 -i td/osd-scrub-repair/1/new.json 2026-02-13T02:26:00.070 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:26:00.079 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-13T02:26:00.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC3i45pa1k8NhAA9l6zTOSbqGVDCHbO4D3sPA== --osd-uuid 7067c579-51ca-42f1-b536-5bd2c1ab0930 2026-02-13T02:26:00.097 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:00.087+0000 7f55e12f3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:26:00.099 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:00.087+0000 7f55e12f3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:26:00.099 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:00.087+0000 7f55e12f3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:26:00.100 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:00.087+0000 7f55e12f3a40 -1 bdev(0x559df6c04c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-13T02:26:00.100 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:00.087+0000 7f55e12f3a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-13T02:26:00.799 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-13T02:26:00.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:26:00.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-13T02:26:00.801 INFO:tasks.workunit.client.0.trial112.stdout:adding osd1 key to auth repository 2026-02-13T02:26:00.801 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:26:00.994 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T02:26:00.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-13T02:26:00.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:26:00.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:26:00.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:26:00.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:26:01.018 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:01.007+0000 7fa8e46a1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:26:01.019 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:01.007+0000 7fa8e46a1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:26:01.021 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:01.011+0000 7fa8e46a1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:26:01.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-13T02:26:01.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:26:01.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:26:01.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:26:01.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:26:01.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:26:01.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:26:01.160 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:26:01.160 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:26:01.160 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:26:01.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:26:01.446 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:01.435+0000 7fa8e46a1a40 -1 Falling back to public interface 2026-02-13T02:26:02.131 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:02.119+0000 7fa8e46a1a40 -1 osd.1 0 log_to_monitors true 2026-02-13T02:26:02.319 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:26:02.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:26:02.320 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:26:02.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:26:02.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:26:02.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:26:02.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:26:03.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:26:03.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:26:03.486 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:26:03.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:26:03.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:26:03.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:26:03.653 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2070880941,v1:127.0.0.1:6811/2070880941] [v2:127.0.0.1:6812/2070880941,v1:127.0.0.1:6813/2070880941] exists,up 7067c579-51ca-42f1-b536-5bd2c1ab0930 2026-02-13T02:26:03.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:26:03.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:26:03.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:26:03.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: unfound_erasure_coded: for id in $(seq 0 3) 2026-02-13T02:26:03.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:919: unfound_erasure_coded: run_osd td/osd-scrub-repair 2 2026-02-13T02:26:03.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:26:03.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:26:03.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-13T02:26:03.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:26:03.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-13T02:26:03.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:26:03.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:26:03.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:26:03.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:26:03.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-13T02:26:03.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-13T02:26:03.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:26:03.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:26:03.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:26:03.658 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:26:03.658 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:26:03.658 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:26:03.658 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:26:03.658 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:26:03.659 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:26:03.659 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:26:03.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:26:03.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:26:03.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:26:03.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:26:03.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:26:03.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:26:03.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:26:03.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:26:03.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:26:03.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:26:03.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:26:03.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:26:03.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-13T02:26:03.660 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:26:03.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c2e2d0ca-6178-4a72-b467-4bbfa8884b36 2026-02-13T02:26:03.661 INFO:tasks.workunit.client.0.trial112.stdout:add osd2 c2e2d0ca-6178-4a72-b467-4bbfa8884b36 2026-02-13T02:26:03.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 c2e2d0ca-6178-4a72-b467-4bbfa8884b36' 2026-02-13T02:26:03.661 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:26:03.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC7i45pJiuWJxAAHl/Az7MUVLa1PgmOWSmErA== 2026-02-13T02:26:03.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC7i45pJiuWJxAAHl/Az7MUVLa1PgmOWSmErA=="}' 2026-02-13T02:26:03.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c2e2d0ca-6178-4a72-b467-4bbfa8884b36 -i td/osd-scrub-repair/2/new.json 2026-02-13T02:26:03.836 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:26:03.845 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-13T02:26:03.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC7i45pJiuWJxAAHl/Az7MUVLa1PgmOWSmErA== --osd-uuid c2e2d0ca-6178-4a72-b467-4bbfa8884b36 2026-02-13T02:26:03.862 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:03.851+0000 7fa2fc96fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:26:03.863 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:03.851+0000 7fa2fc96fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:26:03.865 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:03.855+0000 7fa2fc96fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:26:03.865 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:03.855+0000 7fa2fc96fa40 -1 bdev(0x55a6f6bbac00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-13T02:26:03.865 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:03.855+0000 7fa2fc96fa40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-13T02:26:04.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-13T02:26:04.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:26:04.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-13T02:26:04.493 INFO:tasks.workunit.client.0.trial112.stdout:adding osd2 key to auth repository 2026-02-13T02:26:04.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:26:04.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-13T02:26:04.693 INFO:tasks.workunit.client.0.trial112.stdout:start osd.2 2026-02-13T02:26:04.694 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:26:04.694 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:26:04.694 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:26:04.694 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:26:04.710 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:04.699+0000 7f73e5c1fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:26:04.711 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:04.699+0000 7f73e5c1fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:26:04.712 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:04.703+0000 7f73e5c1fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:26:04.857 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-13T02:26:04.857 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:26:04.857 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T02:26:04.857 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:26:04.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:26:04.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:26:04.858 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:26:04.859 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:26:04.859 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:26:04.859 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:26:05.017 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:26:05.090 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:05.079+0000 7f73e5c1fa40 -1 Falling back to public interface 2026-02-13T02:26:05.662 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:05.651+0000 7f73e5c1fa40 -1 osd.2 0 log_to_monitors true 2026-02-13T02:26:06.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:26:06.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:26:06.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:26:06.019 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:26:06.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:26:06.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:26:06.183 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:26:07.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:26:07.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:26:07.186 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:26:07.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:26:07.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:26:07.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:26:07.350 INFO:tasks.workunit.client.0.trial112.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1452659454,v1:127.0.0.1:6819/1452659454] [v2:127.0.0.1:6820/1452659454,v1:127.0.0.1:6821/1452659454] exists,up c2e2d0ca-6178-4a72-b467-4bbfa8884b36 2026-02-13T02:26:07.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:26:07.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:26:07.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:26:07.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: unfound_erasure_coded: for id in $(seq 0 3) 2026-02-13T02:26:07.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:919: unfound_erasure_coded: run_osd td/osd-scrub-repair 3 2026-02-13T02:26:07.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-13T02:26:07.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:26:07.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-13T02:26:07.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:26:07.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:26:07.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:26:07.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:26:07.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:26:07.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:26:07.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-13T02:26:07.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-13T02:26:07.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:26:07.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:26:07.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:26:07.355 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:26:07.355 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:26:07.355 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:26:07.355 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:26:07.355 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:26:07.355 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:26:07.355 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:26:07.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:26:07.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:26:07.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:26:07.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:26:07.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:26:07.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:26:07.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:26:07.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:26:07.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:26:07.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:26:07.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:26:07.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:26:07.357 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/3 2026-02-13T02:26:07.357 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:26:07.357 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=465184c8-ad13-49ac-912c-fe712508d562 2026-02-13T02:26:07.357 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 465184c8-ad13-49ac-912c-fe712508d562' 2026-02-13T02:26:07.358 INFO:tasks.workunit.client.0.trial112.stdout:add osd3 465184c8-ad13-49ac-912c-fe712508d562 2026-02-13T02:26:07.358 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:26:07.368 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC/i45pQvqEFRAA/zypiMOw2zpsOCfm/KxewQ== 2026-02-13T02:26:07.368 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC/i45pQvqEFRAA/zypiMOw2zpsOCfm/KxewQ=="}' 2026-02-13T02:26:07.368 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 465184c8-ad13-49ac-912c-fe712508d562 -i td/osd-scrub-repair/3/new.json 2026-02-13T02:26:07.533 INFO:tasks.workunit.client.0.trial112.stdout:3 2026-02-13T02:26:07.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/3/new.json 2026-02-13T02:26:07.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC/i45pQvqEFRAA/zypiMOw2zpsOCfm/KxewQ== --osd-uuid 465184c8-ad13-49ac-912c-fe712508d562 2026-02-13T02:26:07.561 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:07.551+0000 7f7901761a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:26:07.563 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:07.551+0000 7f7901761a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:26:07.564 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:07.551+0000 7f7901761a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:26:07.564 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:07.555+0000 7f7901761a40 -1 bdev(0x55752ac76c00 td/osd-scrub-repair/3/block) open stat got: (1) Operation not permitted 2026-02-13T02:26:07.565 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:07.555+0000 7f7901761a40 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2026-02-13T02:26:08.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2026-02-13T02:26:08.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:26:08.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-13T02:26:08.209 INFO:tasks.workunit.client.0.trial112.stdout:adding osd3 key to auth repository 2026-02-13T02:26:08.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:26:08.417 INFO:tasks.workunit.client.0.trial112.stdout:start osd.3 2026-02-13T02:26:08.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-13T02:26:08.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:26:08.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:26:08.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:26:08.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:26:08.434 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:08.423+0000 7f4eb94b0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:26:08.435 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:08.423+0000 7f4eb94b0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:26:08.436 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:08.423+0000 7f4eb94b0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:26:08.574 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-13T02:26:08.575 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:26:08.575 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-13T02:26:08.575 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:26:08.575 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:26:08.575 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:26:08.576 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:26:08.576 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:26:08.576 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:26:08.576 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:26:08.739 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:26:08.850 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:08.839+0000 7f4eb94b0a40 -1 Falling back to public interface 2026-02-13T02:26:09.386 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:09.375+0000 7f4eb94b0a40 -1 osd.3 0 log_to_monitors true 2026-02-13T02:26:09.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:26:09.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:26:09.742 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:26:09.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:26:09.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:26:09.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:26:09.899 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:26:10.901 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:26:10.901 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:26:10.901 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:26:10.901 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:26:10.901 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:26:10.901 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:26:11.068 INFO:tasks.workunit.client.0.trial112.stdout:osd.3 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/4274899235,v1:127.0.0.1:6827/4274899235] [v2:127.0.0.1:6828/4274899235,v1:127.0.0.1:6829/4274899235] exists,up 465184c8-ad13-49ac-912c-fe712508d562 2026-02-13T02:26:11.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:26:11.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:26:11.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:26:11.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: unfound_erasure_coded: create_ec_pool ecpool true k=2 m=2 2026-02-13T02:26:11.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2522: create_ec_pool: local pool_name=ecpool 2026-02-13T02:26:11.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2523: create_ec_pool: shift 2026-02-13T02:26:11.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2524: create_ec_pool: local allow_overwrites=true 2026-02-13T02:26:11.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2525: create_ec_pool: shift 2026-02-13T02:26:11.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2527: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2026-02-13T02:26:11.272 INFO:tasks.workunit.client.0.trial112.stderr:load: isa 2026-02-13T02:26:11.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2529: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2026-02-13T02:26:11.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2026-02-13T02:26:11.576 INFO:tasks.workunit.client.0.trial112.stderr:pool 'ecpool' created 2026-02-13T02:26:11.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:26:12.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2531: create_ec_pool: '[' true = true ']' 2026-02-13T02:26:12.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2532: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2026-02-13T02:26:12.812 INFO:tasks.workunit.client.0.trial112.stderr:set pool 1 allow_ec_overwrites to true 2026-02-13T02:26:12.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2535: create_ec_pool: wait_for_clean 2026-02-13T02:26:12.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:26:12.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:26:12.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:26:12.825 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:26:12.825 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:26:12.825 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:26:12.826 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:26:12.826 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:26:12.826 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:26:12.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:26:12.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:26:12.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:26:12.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:26:12.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:26:12.946 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:26:13.107 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:26:13.108 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:26:13.108 INFO:tasks.workunit.client.0.trial112.stderr:2 2026-02-13T02:26:13.108 INFO:tasks.workunit.client.0.trial112.stderr:3' 2026-02-13T02:26:13.108 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:26:13.108 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:26:13.109 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:26:13.180 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836484 2026-02-13T02:26:13.180 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836484 2026-02-13T02:26:13.180 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484' 2026-02-13T02:26:13.180 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:26:13.180 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:26:13.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705667 2026-02-13T02:26:13.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705667 2026-02-13T02:26:13.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-38654705667' 2026-02-13T02:26:13.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:26:13.257 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:26:13.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542147 2026-02-13T02:26:13.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542147 2026-02-13T02:26:13.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-38654705667 2-60129542147' 2026-02-13T02:26:13.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:26:13.328 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:26:13.402 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411330 2026-02-13T02:26:13.402 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411330 2026-02-13T02:26:13.402 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-38654705667 2-60129542147 3-77309411330' 2026-02-13T02:26:13.402 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:26:13.402 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836484 2026-02-13T02:26:13.403 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:26:13.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:26:13.405 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836484 2026-02-13T02:26:13.405 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:26:13.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836484 2026-02-13T02:26:13.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-13T02:26:13.408 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836484 2026-02-13T02:26:13.408 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:26:13.576 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836484 2026-02-13T02:26:13.576 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:26:14.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:26:14.578 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:26:14.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836484 2026-02-13T02:26:14.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:26:14.745 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705667 2026-02-13T02:26:14.745 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:26:14.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:26:14.747 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705667 2026-02-13T02:26:14.747 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:26:14.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705667 2026-02-13T02:26:14.749 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705667 2026-02-13T02:26:14.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705667' 2026-02-13T02:26:14.750 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:26:14.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705668 -lt 38654705667 2026-02-13T02:26:14.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:26:14.908 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542147 2026-02-13T02:26:14.908 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:26:14.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:26:14.911 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542147 2026-02-13T02:26:14.911 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:26:14.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542147 2026-02-13T02:26:14.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542147' 2026-02-13T02:26:14.913 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542147 2026-02-13T02:26:14.913 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:26:15.079 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542147 -lt 60129542147 2026-02-13T02:26:15.079 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:26:15.080 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-77309411330 2026-02-13T02:26:15.080 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:26:15.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:26:15.082 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-77309411330 2026-02-13T02:26:15.082 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:26:15.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=77309411330 2026-02-13T02:26:15.084 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.3 seq 77309411330 2026-02-13T02:26:15.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 77309411330' 2026-02-13T02:26:15.085 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:26:15.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 77309411330 -lt 77309411330 2026-02-13T02:26:15.251 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:26:15.251 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:26:15.252 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:26:15.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:26:15.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:26:15.445 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:26:15.445 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:26:15.445 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:26:15.445 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:26:15.445 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:26:15.445 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:26:15.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:26:15.604 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:26:15.605 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:26:15.605 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:26:15.797 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:26:15.797 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:26:15.797 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:26:15.797 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2536: create_ec_pool: return 0 2026-02-13T02:26:15.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:924: unfound_erasure_coded: add_something td/osd-scrub-repair ecpool 2026-02-13T02:26:15.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-13T02:26:15.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-13T02:26:15.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2026-02-13T02:26:15.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-13T02:26:15.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-13T02:26:15.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-13T02:26:16.003 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:26:16.017 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-13T02:26:16.208 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:26:16.218 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-13T02:26:16.218 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-13T02:26:16.218 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-13T02:26:16.247 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: unfound_erasure_coded: get_primary ecpool SOMETHING 2026-02-13T02:26:16.248 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2026-02-13T02:26:16.248 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-13T02:26:16.248 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T02:26:16.248 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2026-02-13T02:26:16.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: unfound_erasure_coded: local primary=1 2026-02-13T02:26:16.415 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:927: unfound_erasure_coded: get_osds ecpool SOMETHING 2026-02-13T02:26:16.415 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2026-02-13T02:26:16.415 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:927: unfound_erasure_coded: sed -e s/1// 2026-02-13T02:26:16.415 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-13T02:26:16.416 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2026-02-13T02:26:16.416 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-13T02:26:16.581 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-13T02:26:16.581 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:26:16.581 INFO:tasks.workunit.client.0.trial112.stderr:2 2026-02-13T02:26:16.581 INFO:tasks.workunit.client.0.trial112.stderr:3' 2026-02-13T02:26:16.581 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2 3 2026-02-13T02:26:16.585 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:927: unfound_erasure_coded: osds=('0' '2' '3') 2026-02-13T02:26:16.585 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:927: unfound_erasure_coded: local -a osds 2026-02-13T02:26:16.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:928: unfound_erasure_coded: local not_primary_first=0 2026-02-13T02:26:16.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:929: unfound_erasure_coded: local not_primary_second=2 2026-02-13T02:26:16.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:930: unfound_erasure_coded: local not_primary_third=3 2026-02-13T02:26:16.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: unfound_erasure_coded: pids= 2026-02-13T02:26:16.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2026-02-13T02:26:16.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-13T02:26:16.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-13T02:26:16.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 448754"' 2026-02-13T02:26:16.586 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 448754' 2026-02-13T02:26:16.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:937: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2026-02-13T02:26:16.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-13T02:26:16.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-13T02:26:16.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 448755"' 2026-02-13T02:26:16.587 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 448755' 2026-02-13T02:26:16.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:938: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-13T02:26:16.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-13T02:26:16.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-13T02:26:16.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 448757"' 2026-02-13T02:26:16.587 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 448757' 2026-02-13T02:26:16.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: unfound_erasure_coded: wait_background pids 2026-02-13T02:26:16.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 448754 448755 448757' 2026-02-13T02:26:16.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-13T02:26:16.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-13T02:26:16.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 448754 2026-02-13T02:26:16.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2026-02-13T02:26:16.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/448758: /' 2026-02-13T02:26:16.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2026-02-13T02:26:16.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/448760: /' 2026-02-13T02:26:16.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-13T02:26:16.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/448762: /' 2026-02-13T02:26:17.258 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:26:17.258 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:26:17.258 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-13T02:26:17.258 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:26:17.258 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2026-02-13T02:26:17.258 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:26:17.258 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:26:17.259 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-13T02:26:17.259 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:26:17.259 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-13T02:26:17.259 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:26:17.259 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:26:17.259 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:26:17.259 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:26:17.259 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:26:17.259 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:26:17.259 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2026-02-13T02:26:17.259 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:26:17.260 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:26:17.260 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-13T02:26:17.260 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:26:17.260 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:26:17.260 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2026-02-13T02:26:17.260 INFO:tasks.workunit.client.0.trial112.stderr:448762: remove 3#1:eb822e21:::SOMETHING:head# 2026-02-13T02:26:17.260 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-13T02:26:17.260 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:26:17.260 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:26:17.260 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-13T02:26:17.260 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:26:17.261 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-13T02:26:17.261 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:26:17.261 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:26:17.261 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: 448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:26:17.261 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:26:17.261 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-13T02:26:17.261 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:26:17.261 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2026-02-13T02:26:17.261 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:26:17.262 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:26:17.262 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-13T02:26:17.262 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:26:17.262 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-13T02:26:17.262 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:26:17.262 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:26:17.262 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:26:17.262 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:26:17.262 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:26:17.262 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:26:17.262 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2026-02-13T02:26:17.263 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:26:17.263 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:26:17.263 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-13T02:26:17.263 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:26:17.263 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:26:17.263 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2026-02-13T02:26:17.263 INFO:tasks.workunit.client.0.trial112.stderr:448758: remove 1#1:eb822e21:::SOMETHING:head# 2026-02-13T02:26:17.263 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-13T02:26:17.263 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:26:17.263 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:26:17.263 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:26:17.264 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:26:17.264 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-13T02:26:17.264 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:26:17.264 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:26:17.265 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:26:17.265 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:26:17.265 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-13T02:26:17.265 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-13T02:26:17.265 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:26:17.265 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:26:17.265 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:26:17.265 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:26:17.266 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:26:17.266 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:26:17.266 INFO:tasks.workunit.client.0.trial112.stderr:448762: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:26:17.266 INFO:tasks.workunit.client.0.trial112.stderr:448762: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:26:17.266 INFO:tasks.workunit.client.0.trial112.stderr:448762: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:26:17.266 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:26:17.266 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:26:17.266 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:26:17.266 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:26:17.266 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:26:17.266 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:26:17.266 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:26:17.267 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:26:17.267 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:26:17.267 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:26:17.267 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:26:17.267 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-13T02:26:17.267 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-13T02:26:17.267 INFO:tasks.workunit.client.0.trial112.stderr:448762: start osd.3 2026-02-13T02:26:17.267 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:26:17.267 INFO:tasks.workunit.client.0.trial112.stderr:4activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:26:17.267 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:26:17.268 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-13T02:26:17.268 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-13T02:26:17.268 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:26:17.268 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:26:17.268 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:26:17.268 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:26:17.268 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:26:17.268 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:26:17.268 INFO:tasks.workunit.client.0.trial112.stderr:448758: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:26:17.268 INFO:tasks.workunit.client.0.trial112.stderr:448758: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:26:17.269 INFO:tasks.workunit.client.0.trial112.stderr:448758: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:26:17.269 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:26:17.269 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:26:17.269 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:26:17.269 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:26:17.269 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:26:17.269 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:26:17.269 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:26:17.269 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:26:17.269 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:26:17.270 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:26:17.270 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:26:17.270 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-13T02:26:17.270 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:26:17.270 INFO:tasks.workunit.client.0.trial112.stderr:448758: start osd.0 2026-02-13T02:26:17.270 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:26:17.270 INFO:tasks.workunit.client.0.trial112.stderr:4448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-13T02:26:17.271 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:26:17.271 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-13T02:26:17.271 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:26:17.271 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2026-02-13T02:26:17.271 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-13T02:26:17.271 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:26:17.271 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-13T02:26:17.271 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:26:17.271 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2026-02-13T02:26:17.272 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:26:17.272 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:26:17.272 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:26:17.272 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:26:17.272 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:26:17.272 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:26:17.272 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2026-02-13T02:26:17.272 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-13T02:26:17.272 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:26:17.272 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-13T02:26:17.272 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:26:17.273 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2026-02-13T02:26:17.273 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2026-02-13T02:26:17.273 INFO:tasks.workunit.client.0.trial112.stderr:448760: remove 2#1:eb822e21:::SOMETHING:head# 2026-02-13T02:26:17.273 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2026-02-13T02:26:17.273 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-13T02:26:17.273 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:26:17.273 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-13T02:26:17.273 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:26:17.273 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2026-02-13T02:26:17.273 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-13T02:26:17.274 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:26:17.274 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:26:17.274 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:26:17.274 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-13T02:26:17.274 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-13T02:26:17.274 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:26:17.275 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:26:17.275 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-13T02:26:17.275 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:26:17.275 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:26:17.275 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:26:17.275 INFO:tasks.workunit.client.0.trial112.stderr:448760: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:26:17.275 INFO:tasks.workunit.client.0.trial112.stderr:448760: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:26:17.275 INFO:tasks.workunit.client.0.trial112.stderr:448760: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:26:17.275 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:26:17.275 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' 2026-02-13T02:26:17.276 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:26:17.276 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-13T02:26:17.276 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-13T02:26:17.276 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:26:17.276 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:26:17.276 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:26:17.276 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:26:17.276 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:26:17.276 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:26:17.276 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2026-02-13T02:26:17.276 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-13T02:26:17.277 INFO:tasks.workunit.client.0.trial112.stderr:448760: start osd.2 2026-02-13T02:26:17.277 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=bc99bd57-8f80-4bf5-850a-1582db87f998 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.35207/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:26:18.759 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-13T02:26:18.759 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-13T02:26:18.759 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:26:18.759 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:26:18.759 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:26:18.759 INFO:tasks.workunit.client.0.trial112.stderr:448762: 2026-02-13T02:26:17.267+0000 7f9033bbea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:26:18.759 INFO:tasks.workunit.client.0.trial112.stderr:448762: 2026-02-13T02:26:17.267+0000 7f9033bbea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:26:18.759 INFO:tasks.workunit.client.0.trial112.stderr:448762: 2026-02-13T02:26:17.267+0000 7f9033bbea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:26:18.760 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-13T02:26:18.760 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:26:18.760 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-13T02:26:18.760 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:26:18.760 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:26:18.760 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:26:18.760 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:26:18.760 INFO:tasks.workunit.client.0.trial112.stderr:448762: 0 2026-02-13T02:26:18.760 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:26:18.760 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:26:18.760 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:26:18.761 INFO:tasks.workunit.client.0.trial112.stderr:448762: 2026-02-13T02:26:17.747+0000 7f9033bbea40 -1 Falling back to public interface 2026-02-13T02:26:18.761 INFO:tasks.workunit.client.0.trial112.stderr:448762: 2026-02-13T02:26:18.479+0000 7f9033bbea40 -1 osd.3 26 log_to_monitors true 2026-02-13T02:26:18.761 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:26:18.761 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:26:18.761 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:26:18.761 INFO:tasks.workunit.client.0.trial112.stderr:448762: 1 2026-02-13T02:26:18.761 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:26:18.761 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-13T02:26:18.761 INFO:tasks.workunit.client.0.trial112.stderr:448762: osd.3 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [18,26) [v2:127.0.0.1:6800/2672171282,v1:127.0.0.1:6801/2672171282] [v2:127.0.0.1:6802/2672171282,v1:127.0.0.1:6803/2672171282] exists,up 465184c8-ad13-49ac-912c-fe712508d562 2026-02-13T02:26:18.761 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:26:18.761 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:26:18.762 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:26:18.762 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:26:18.762 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:26:18.762 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:26:18.762 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:26:18.762 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:26:18.762 INFO:tasks.workunit.client.0.trial112.stderr:448762: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:26:19.351 INFO:tasks.workunit.client.0.trial112.stderr:448762: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:26:19.351 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:26:19.351 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:26:19.351 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:26:19.352 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:26:19.352 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:26:19.352 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:26:19.352 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:26:19.352 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:26:19.352 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:26:19.352 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:26:19.352 INFO:tasks.workunit.client.0.trial112.stderr:448762: 1 2026-02-13T02:26:19.352 INFO:tasks.workunit.client.0.trial112.stderr:448762: 2 2026-02-13T02:26:19.352 INFO:tasks.workunit.client.0.trial112.stderr:448762: 3' 2026-02-13T02:26:19.352 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:26:19.352 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:26:19.353 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:26:19.353 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986178 2026-02-13T02:26:19.353 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986178 2026-02-13T02:26:19.353 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-133143986178' 2026-02-13T02:26:19.353 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:26:19.353 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:26:19.353 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705670 2026-02-13T02:26:19.353 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705670 2026-02-13T02:26:19.353 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-133143986178 1-38654705670' 2026-02-13T02:26:19.353 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:26:19.353 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:26:19.353 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986178 2026-02-13T02:26:19.354 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986178 2026-02-13T02:26:19.354 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-133143986178 1-38654705670 2-133143986178' 2026-02-13T02:26:19.354 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:26:19.354 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:26:19.354 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018882 2026-02-13T02:26:19.354 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018882 2026-02-13T02:26:19.354 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-133143986178 1-38654705670 2-133143986178 3-128849018882' 2026-02-13T02:26:19.354 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:26:19.354 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-133143986178 2026-02-13T02:26:19.921 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home48758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-13T02:26:19.921 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:26:19.922 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:26:19.922 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:26:19.922 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:26:19.922 INFO:tasks.workunit.client.0.trial112.stderr:448758: 2026-02-13T02:26:17.267+0000 7fdfd2a43a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:26:19.922 INFO:tasks.workunit.client.0.trial112.stderr:448758: 2026-02-13T02:26:17.271+0000 7fdfd2a43a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:26:19.922 INFO:tasks.workunit.client.0.trial112.stderr:448758: 2026-02-13T02:26:17.271+0000 7fdfd2a43a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:26:19.922 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:26:19.922 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:26:19.922 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:26:19.922 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:26:19.922 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:26:19.922 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:26:19.923 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:26:19.923 INFO:tasks.workunit.client.0.trial112.stderr:448758: 0 2026-02-13T02:26:19.923 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:26:19.923 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:26:19.923 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:26:19.923 INFO:tasks.workunit.client.0.trial112.stderr:448758: 2026-02-13T02:26:17.771+0000 7fdfd2a43a40 -1 Falling back to public interface 2026-02-13T02:26:19.923 INFO:tasks.workunit.client.0.trial112.stderr:448758: 2026-02-13T02:26:18.551+0000 7fdfd2a43a40 -1 osd.0 26 log_to_monitors true 2026-02-13T02:26:19.923 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:26:19.923 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:26:19.923 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:26:19.923 INFO:tasks.workunit.client.0.trial112.stderr:448758: 1 2026-02-13T02:26:19.923 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:26:19.924 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:26:19.924 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:26:19.924 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:26:19.924 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:26:19.924 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:26:19.924 INFO:tasks.workunit.client.0.trial112.stderr:448758: 2 2026-02-13T02:26:19.924 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:26:19.924 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:26:19.924 INFO:tasks.workunit.client.0.trial112.stderr:448758: osd.0 up in weight 1 up_from 31 up_thru 0 down_at 27 last_clean_interval [5,26) [v2:127.0.0.1:6818/583890008,v1:127.0.0.1:6819/583890008] [v2:127.0.0.1:6820/583890008,v1:127.0.0.1:6821/583890008] exists,up 339d5a6c-c266-4659-9a1d-9fee970374d4 2026-02-13T02:26:19.924 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:26:19.924 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:26:19.924 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:26:19.925 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:26:19.931 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client48760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2026-02-13T02:26:19.931 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-13T02:26:19.931 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:26:19.931 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:26:19.931 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:26:19.931 INFO:tasks.workunit.client.0.trial112.stderr:448760: 2026-02-13T02:26:17.279+0000 7f5d63394a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:26:19.931 INFO:tasks.workunit.client.0.trial112.stderr:448760: 2026-02-13T02:26:17.283+0000 7f5d63394a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:26:19.931 INFO:tasks.workunit.client.0.trial112.stderr:448760: 2026-02-13T02:26:17.283+0000 7f5d63394a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:26:19.931 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-13T02:26:19.932 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:26:19.932 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T02:26:19.932 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:26:19.932 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:26:19.932 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:26:19.932 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:26:19.932 INFO:tasks.workunit.client.0.trial112.stderr:448760: 0 2026-02-13T02:26:19.932 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:26:19.932 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:26:19.932 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:26:19.932 INFO:tasks.workunit.client.0.trial112.stderr:448760: 2026-02-13T02:26:17.803+0000 7f5d63394a40 -1 Falling back to public interface 2026-02-13T02:26:19.932 INFO:tasks.workunit.client.0.trial112.stderr:448760: 2026-02-13T02:26:18.571+0000 7f5d63394a40 -1 osd.2 26 log_to_monitors true 2026-02-13T02:26:19.932 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:26:19.933 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:26:19.933 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:26:19.933 INFO:tasks.workunit.client.0.trial112.stderr:448760: 1 2026-02-13T02:26:19.933 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:26:19.933 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:26:19.933 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:26:19.933 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:26:19.933 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:26:19.933 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:26:19.933 INFO:tasks.workunit.client.0.trial112.stderr:448760: 2 2026-02-13T02:26:19.933 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:26:19.933 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:26:19.934 INFO:tasks.workunit.client.0.trial112.stderr:448760: osd.2 up in weight 1 up_from 31 up_thru 0 down_at 27 last_clean_interval [14,26) [v2:127.0.0.1:6826/2868812768,v1:127.0.0.1:6827/2868812768] [v2:127.0.0.1:6828/2868812768,v1:127.0.0.1:6829/2868812768] exists,up c2e2d0ca-6178-4a72-b467-4bbfa8884b36 2026-02-13T02:26:19.934 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:26:19.934 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:26:19.934 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:26:19.934 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:26:20.432 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:26:20.432 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:26:20.432 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:26:20.432 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:26:20.432 INFO:tasks.workunit.client.0.trial112.stderr:448760: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:26:20.432 INFO:tasks.workunit.client.0.trial112.stderr:448760: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:26:20.433 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:26:20.433 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:26:20.433 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:26:20.433 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:26:20.433 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:26:20.433 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:26:20.433 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:26:20.433 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:26:20.433 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:26:20.433 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:26:20.433 INFO:tasks.workunit.client.0.trial112.stderr:448760: 1 2026-02-13T02:26:20.433 INFO:tasks.workunit.client.0.trial112.stderr:448760: 2 2026-02-13T02:26:20.434 INFO:tasks.workunit.client.0.trial112.stderr:448760: 3' 2026-02-13T02:26:20.434 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:26:20.434 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:26:20.434 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:26:20.434 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986180 2026-02-13T02:26:20.434 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986180 2026-02-13T02:26:20.434 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-133143986180' 2026-02-13T02:26:20.434 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:26:20.434 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:26:20.434 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705672 2026-02-13T02:26:20.434 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705672 2026-02-13T02:26:20.434 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-133143986180 1-38654705672' 2026-02-13T02:26:20.435 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:26:20.435 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:26:20.435 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986180 2026-02-13T02:26:20.435 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986180 2026-02-13T02:26:20.435 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-133143986180 1-38654705672 2-133143986180' 2026-02-13T02:26:20.435 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:26:20.485 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_st.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:26:20.485 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:26:20.485 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:26:20.486 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:26:20.486 INFO:tasks.workunit.client.0.trial112.stderr:448758: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:26:20.486 INFO:tasks.workunit.client.0.trial112.stderr:448758: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:26:20.486 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:26:20.486 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:26:20.486 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:26:20.486 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:26:20.486 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:26:20.486 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:26:20.486 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:26:20.486 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:26:20.486 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:26:20.487 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:26:20.487 INFO:tasks.workunit.client.0.trial112.stderr:448758: 1 2026-02-13T02:26:20.487 INFO:tasks.workunit.client.0.trial112.stderr:448758: 2 2026-02-13T02:26:20.487 INFO:tasks.workunit.client.0.trial112.stderr:448758: 3' 2026-02-13T02:26:20.487 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:26:20.487 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:26:20.487 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:26:20.487 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986179 2026-02-13T02:26:20.487 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986179 2026-02-13T02:26:20.487 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-133143986179' 2026-02-13T02:26:20.487 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:26:20.487 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:26:20.488 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705671 2026-02-13T02:26:20.488 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705671 2026-02-13T02:26:20.488 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-133143986179 1-38654705671' 2026-02-13T02:26:20.488 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:26:20.488 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:26:20.488 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986179 2026-02-13T02:26:20.488 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986179 2026-02-13T02:26:20.488 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-133143986179 1-38654705671 2-133143986179' 2026-02-13T02:26:20.488 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:26:20.488 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-13T02:26:21.998 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018883 2026-02-13T02:26:21.998 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018883 2026-02-13T02:26:21.998 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-133143986179 1-38654705671 2-133143986179 3-128849018883' 2026-02-13T02:26:21.998 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:26:21.998 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-133143986179 2026-02-13T02:26:21.998 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:26:21.998 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:26:21.999 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-133143986179 2026-02-13T02:26:21.999 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:26:21.999 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=133143986179 2026-02-13T02:26:21.999 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 133143986179' 2026-02-13T02:26:21.999 INFO:tasks.workunit.client.0.trial112.stderr:448758: waiting osd.0 seq 133143986179 2026-02-13T02:26:21.999 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:26:21.999 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 133143986179 2026-02-13T02:26:21.999 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:26:21.999 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:26:21.999 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:26:21.999 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 133143986180 -lt 133143986179 2026-02-13T02:26:21.999 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:26:22.000 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705671 2026-02-13T02:26:22.000 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:26:22.000 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:26:22.000 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705671 2026-02-13T02:26:22.000 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:26:22.000 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705671 2026-02-13T02:26:22.000 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705671' 2026-02-13T02:26:22.000 INFO:tasks.workunit.client.0.trial112.stderr:448758: waiting osd.1 seq 38654705671 2026-02-13T02:26:22.000 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:26:22.000 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705672 -lt 38654705671 2026-02-13T02:26:22.000 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:26:22.000 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-133143986179 2026-02-13T02:26:22.000 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:26:22.001 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:26:22.001 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-133143986179 2026-02-13T02:26:22.001 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:26:22.010 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_ats 2026-02-13T02:26:22.010 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018884 2026-02-13T02:26:22.010 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018884 2026-02-13T02:26:22.010 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-133143986180 1-38654705672 2-133143986180 3-128849018884' 2026-02-13T02:26:22.010 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:26:22.010 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-133143986180 2026-02-13T02:26:22.010 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:26:22.011 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:26:22.011 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-133143986180 2026-02-13T02:26:22.011 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:26:22.011 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=133143986180 2026-02-13T02:26:22.011 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 133143986180' 2026-02-13T02:26:22.011 INFO:tasks.workunit.client.0.trial112.stderr:448760: waiting osd.0 seq 133143986180 2026-02-13T02:26:22.011 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:26:22.011 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 133143986180 2026-02-13T02:26:22.011 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:26:22.011 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:26:22.011 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:26:22.011 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 133143986180 -lt 133143986180 2026-02-13T02:26:22.012 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:26:22.012 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705672 2026-02-13T02:26:22.012 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:26:22.012 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:26:22.012 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705672 2026-02-13T02:26:22.012 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:26:22.012 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705672 2026-02-13T02:26:22.012 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705672' 2026-02-13T02:26:22.012 INFO:tasks.workunit.client.0.trial112.stderr:448760: waiting osd.1 seq 38654705672 2026-02-13T02:26:22.012 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:26:22.012 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705672 -lt 38654705672 2026-02-13T02:26:22.013 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:26:22.013 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-133143986180 2026-02-13T02:26:22.013 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:26:22.013 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:26:22.013 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-133143986180 2026-02-13T02:26:22.013 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:26:22.025 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flus/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:26:22.025 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:26:22.025 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-133143986178 2026-02-13T02:26:22.025 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:26:22.025 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=133143986178 2026-02-13T02:26:22.025 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 133143986178' 2026-02-13T02:26:22.025 INFO:tasks.workunit.client.0.trial112.stderr:448762: waiting osd.0 seq 133143986178 2026-02-13T02:26:22.026 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:26:22.026 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 133143986178 2026-02-13T02:26:22.026 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:26:22.026 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:26:22.026 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:26:22.026 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 133143986178 2026-02-13T02:26:22.026 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:26:22.026 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:26:22.026 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:26:22.026 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 133143986180 -lt 133143986178 2026-02-13T02:26:22.026 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:26:22.026 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705670 2026-02-13T02:26:22.026 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:26:22.027 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:26:22.027 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705670 2026-02-13T02:26:22.027 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:26:22.027 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705670 2026-02-13T02:26:22.027 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705670' 2026-02-13T02:26:22.027 INFO:tasks.workunit.client.0.trial112.stderr:448762: waiting osd.1 seq 38654705670 2026-02-13T02:26:22.027 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:26:22.027 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705672 -lt 38654705670 2026-02-13T02:26:22.027 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:26:22.027 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-133143986178 2026-02-13T02:26:22.027 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:26:22.027 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:26:22.027 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-133143986178 2026-02-13T02:26:22.028 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:26:22.028 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=133143986178 2026-02-13T02:26:22.028 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 133143986178' 2026-02-13T02:26:22.877 INFO:tasks.workunit.client.0.trial112.stderr:44876stats: seq=133143986179 2026-02-13T02:26:22.878 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 133143986179' 2026-02-13T02:26:22.878 INFO:tasks.workunit.client.0.trial112.stderr:448758: waiting osd.2 seq 133143986179 2026-02-13T02:26:22.878 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:26:22.878 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 133143986180 -lt 133143986179 2026-02-13T02:26:22.878 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:26:22.878 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-128849018883 2026-02-13T02:26:22.878 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:26:22.878 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:26:22.879 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-128849018883 2026-02-13T02:26:22.879 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:26:22.879 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018883 2026-02-13T02:26:22.879 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 128849018883' 2026-02-13T02:26:22.879 INFO:tasks.workunit.client.0.trial112.stderr:448758: waiting osd.3 seq 128849018883 2026-02-13T02:26:22.879 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:26:22.879 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018884 -lt 128849018883 2026-02-13T02:26:22.879 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:26:22.879 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:26:22.879 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:26:22.879 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:26:22.879 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:26:22.880 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:26:22.880 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:26:22.880 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:26:22.880 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:26:22.880 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:26:22.880 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:26:22.880 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:26:22.880 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:26:22.880 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:26:22.880 INFO:tasks.workunit.client.0.trial112.stderr:448758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:26:22.880 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:26:22.880 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:26:22.881 INFO:tasks.workunit.client.0.trial112.stderr:448758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:26:22.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-13T02:26:22.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-13T02:26:22.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 448755 2026-02-13T02:26:22.901 INFO:tasks.workunit.client.0.trial112.stderr:h_pg_stats: seq=133143986180 2026-02-13T02:26:22.901 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 133143986180' 2026-02-13T02:26:22.901 INFO:tasks.workunit.client.0.trial112.stderr:448760: waiting osd.2 seq 133143986180 2026-02-13T02:26:22.901 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:26:22.901 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 133143986180 -lt 133143986180 2026-02-13T02:26:22.901 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:26:22.902 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-128849018884 2026-02-13T02:26:22.902 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:26:22.902 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:26:22.902 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-128849018884 2026-02-13T02:26:22.902 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:26:22.902 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018884 2026-02-13T02:26:22.902 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 128849018884' 2026-02-13T02:26:22.902 INFO:tasks.workunit.client.0.trial112.stderr:448760: waiting osd.3 seq 128849018884 2026-02-13T02:26:22.902 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:26:22.902 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018884 -lt 128849018884 2026-02-13T02:26:22.902 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:26:22.902 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:26:22.903 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:26:22.903 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:26:22.903 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:26:22.903 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:26:22.903 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:26:22.903 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:26:22.903 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:26:22.903 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:26:22.903 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:26:22.903 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:26:22.903 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:26:22.903 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:26:22.903 INFO:tasks.workunit.client.0.trial112.stderr:448760: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:26:22.904 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:26:22.904 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:26:22.904 INFO:tasks.workunit.client.0.trial112.stderr:448760: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:26:22.904 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-13T02:26:22.904 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-13T02:26:22.904 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 448757 2026-02-13T02:26:22.924 INFO:tasks.workunit.client.0.trial112.stderr:2: waiting osd.2 seq 133143986178 2026-02-13T02:26:22.924 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:26:22.925 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 133143986180 -lt 133143986178 2026-02-13T02:26:22.925 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:26:22.925 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-128849018882 2026-02-13T02:26:22.925 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:26:22.925 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-13T02:26:22.925 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-128849018882 2026-02-13T02:26:22.925 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:26:22.925 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018882 2026-02-13T02:26:22.925 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 128849018882' 2026-02-13T02:26:22.925 INFO:tasks.workunit.client.0.trial112.stderr:448762: waiting osd.3 seq 128849018882 2026-02-13T02:26:22.925 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-13T02:26:22.926 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018884 -lt 128849018882 2026-02-13T02:26:22.926 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:26:22.926 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:26:22.926 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:26:22.926 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:26:22.926 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:26:22.926 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:26:22.926 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:26:22.926 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:26:22.926 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:26:22.926 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:26:22.926 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:26:22.927 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:26:22.927 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:26:22.927 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:26:22.927 INFO:tasks.workunit.client.0.trial112.stderr:448762: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:26:22.927 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:26:22.927 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:26:22.927 INFO:tasks.workunit.client.0.trial112.stderr:448762: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:26:22.927 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-13T02:26:22.927 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-13T02:26:22.927 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-13T02:26:22.927 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-13T02:26:22.927 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: unfound_erasure_coded: return_code=0 2026-02-13T02:26:22.927 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:941: unfound_erasure_coded: '[' 0 -ne 0 ']' 2026-02-13T02:26:22.928 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: unfound_erasure_coded: get_pg ecpool SOMETHING 2026-02-13T02:26:22.928 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-13T02:26:22.929 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-13T02:26:22.929 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-13T02:26:22.929 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T02:26:23.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: unfound_erasure_coded: local pg=1.0 2026-02-13T02:26:23.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:947: unfound_erasure_coded: repair 1.0 2026-02-13T02:26:23.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=1.0 2026-02-13T02:26:23.092 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 1.0 2026-02-13T02:26:23.092 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:26:23.093 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:26:23.093 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:26:23.093 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:26:23.254 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-13T02:26:11.569090+0000 2026-02-13T02:26:23.255 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 1.0 2026-02-13T02:26:23.403 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 1.0s0 on osd.1 to repair 2026-02-13T02:26:23.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 1.0 2026-02-13T02:26:11.569090+0000 2026-02-13T02:26:23.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-13T02:26:23.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:26:11.569090+0000 2026-02-13T02:26:23.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:26:23.414 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:26:23.414 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:26:23.414 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:26:23.415 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:26:23.415 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:26:23.415 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:26:23.415 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:26:23.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:26:11.569090+0000 '>' 2026-02-13T02:26:11.569090+0000 2026-02-13T02:26:23.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:26:24.580 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:26:24.581 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:26:24.581 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:26:24.581 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:26:24.582 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:26:24.582 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:26:24.582 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:26:24.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:26:11.569090+0000 '>' 2026-02-13T02:26:11.569090+0000 2026-02-13T02:26:24.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:26:25.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:26:25.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:26:25.746 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:26:25.746 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:26:25.746 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:26:25.747 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:26:25.747 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:26:25.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:26:11.569090+0000 '>' 2026-02-13T02:26:11.569090+0000 2026-02-13T02:26:25.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:26:26.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:26:26.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:26:26.913 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:26:26.913 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:26:26.913 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:26:26.913 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:26:26.914 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:26:27.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:26:11.569090+0000 '>' 2026-02-13T02:26:11.569090+0000 2026-02-13T02:26:27.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:26:28.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:26:28.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:26:28.080 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:26:28.081 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:26:28.081 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:26:28.081 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:26:28.081 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:26:28.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:26:11.569090+0000 '>' 2026-02-13T02:26:11.569090+0000 2026-02-13T02:26:28.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:26:29.251 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:26:29.251 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:26:29.252 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:26:29.252 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:26:29.252 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:26:29.253 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:26:29.253 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:26:29.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:26:24.075158+0000 '>' 2026-02-13T02:26:11.569090+0000 2026-02-13T02:26:29.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:26:29.417 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:952: unfound_erasure_coded: seq 1 60 2026-02-13T02:26:29.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:952: unfound_erasure_coded: for f in `seq 1 60` 2026-02-13T02:26:29.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: unfound_erasure_coded: ceph -s 2026-02-13T02:26:29.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: unfound_erasure_coded: grep '1/1 objects unfound' 2026-02-13T02:26:29.613 INFO:tasks.workunit.client.0.trial112.stdout: 1/1 objects unfound (100.000%) 2026-02-13T02:26:29.613 INFO:tasks.workunit.client.0.trial112.stdout: 1/1 objects unfound (100.000%) 2026-02-13T02:26:29.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: unfound_erasure_coded: break 2026-02-13T02:26:29.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:956: unfound_erasure_coded: ceph -s 2026-02-13T02:26:29.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:956: unfound_erasure_coded: grep '4 up' 2026-02-13T02:26:29.818 INFO:tasks.workunit.client.0.trial112.stdout: osd: 4 osds: 4 up (since 11s), 4 in (since 22s) 2026-02-13T02:26:29.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: unfound_erasure_coded: ceph -s 2026-02-13T02:26:29.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: unfound_erasure_coded: grep '4 in' 2026-02-13T02:26:30.017 INFO:tasks.workunit.client.0.trial112.stdout: osd: 4 osds: 4 up (since 11s), 4 in (since 22s) 2026-02-13T02:26:30.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: unfound_erasure_coded: ceph -s 2026-02-13T02:26:30.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: unfound_erasure_coded: grep '1/1 objects unfound' 2026-02-13T02:26:30.213 INFO:tasks.workunit.client.0.trial112.stdout: 1/1 objects unfound (100.000%) 2026-02-13T02:26:30.213 INFO:tasks.workunit.client.0.trial112.stdout: 1/1 objects unfound (100.000%) 2026-02-13T02:26:30.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-13T02:26:30.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:26:30.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:26:30.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:26:30.214 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:26:30.214 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:26:30.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:26:30.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:26:30.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:26:30.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:26:30.328 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:26:30.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:26:30.329 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:26:30.331 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:26:30.331 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:26:30.331 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:26:30.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:26:30.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:26:30.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:26:30.334 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:26:30.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:26:30.336 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:26:30.336 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:26:30.348 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:26:30.348 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:26:30.348 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:26:30.348 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:26:30.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:26:30.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:26:30.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: main: code=0 2026-02-13T02:26:30.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: teardown td/osd-scrub-repair 0 2026-02-13T02:26:30.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-13T02:26:30.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-13T02:26:30.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-13T02:26:30.350 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:26:30.350 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:26:30.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:26:30.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:26:30.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:26:30.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:26:30.354 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:26:30.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:26:30.356 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:26:30.357 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:26:30.357 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:26:30.358 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:26:30.359 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:26:30.359 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:26:30.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:26:30.360 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:26:30.361 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:26:30.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-13T02:26:30.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-13T02:26:30.364 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:26:30.364 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:26:30.364 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35207 2026-02-13T02:26:30.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35207 2026-02-13T02:26:30.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:26:30.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:26:30.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: main: return 0 2026-02-13T02:26:30.366 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-13T02:26:30.366 DEBUG:teuthology.orchestra.run.trial112:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-13T02:26:30.419 INFO:tasks.workunit:Running workunit scrub/osd-scrub-snaps.sh... 2026-02-13T02:26:30.420 DEBUG:teuthology.orchestra.run.trial112:workunit test scrub/osd-scrub-snaps.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh 2026-02-13T02:26:30.469 INFO:tasks.workunit.client.0.trial112.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-13T02:26:30.472 INFO:tasks.workunit.client.0.trial112.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-13T02:26:30.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-13T02:26:30.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-13T02:26:30.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export PYTHONWARNINGS=ignore 2026-02-13T02:26:30.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: PYTHONWARNINGS=ignore 2026-02-13T02:26:30.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: export CEPH_CONF=/dev/null 2026-02-13T02:26:30.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: CEPH_CONF=/dev/null 2026-02-13T02:26:30.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: main: unset CEPH_ARGS 2026-02-13T02:26:30.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: local code 2026-02-13T02:26:30.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: run td/osd-scrub-snaps 2026-02-13T02:26:30.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:27: run: local dir=td/osd-scrub-snaps 2026-02-13T02:26:30.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:28: run: shift 2026-02-13T02:26:30.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:30: run: export CEPH_MON=127.0.0.1:7121 2026-02-13T02:26:30.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:30: run: CEPH_MON=127.0.0.1:7121 2026-02-13T02:26:30.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:31: run: export CEPH_ARGS 2026-02-13T02:26:30.473 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:32: run: uuidgen 2026-02-13T02:26:30.474 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:32: run: CEPH_ARGS+='--fsid=b2b95964-f5a3-4b20-bd2b-d60f1ae5f8a0 --auth-supported=none ' 2026-02-13T02:26:30.474 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:33: run: CEPH_ARGS+='--mon-host=127.0.0.1:7121 ' 2026-02-13T02:26:30.474 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:35: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2026-02-13T02:26:30.474 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:36: run: set 2026-02-13T02:26:30.474 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:36: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-13T02:26:30.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:36: run: local 'funcs=TEST_scrub_snaps 2026-02-13T02:26:30.477 INFO:tasks.workunit.client.0.trial112.stderr:TEST_scrub_snaps_primary 2026-02-13T02:26:30.477 INFO:tasks.workunit.client.0.trial112.stderr:TEST_scrub_snaps_replica' 2026-02-13T02:26:30.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:37: run: for func in $funcs 2026-02-13T02:26:30.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:38: run: setup td/osd-scrub-snaps 2026-02-13T02:26:30.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-snaps 2026-02-13T02:26:30.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-snaps 2026-02-13T02:26:30.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2026-02-13T02:26:30.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:26:30.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2026-02-13T02:26:30.477 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:26:30.477 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:26:30.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:26:30.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:26:30.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:26:30.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:26:30.480 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:26:30.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:26:30.481 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:26:30.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:26:30.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:26:30.483 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:26:30.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:26:30.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:26:30.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:26:30.484 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:26:30.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:26:30.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:26:30.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2026-02-13T02:26:30.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:26:30.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:26:30.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.453125 2026-02-13T02:26:30.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.453125 2026-02-13T02:26:30.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:26:30.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:26:30.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-snaps 2026-02-13T02:26:30.491 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T02:26:30.491 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:26:30.491 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.453125 2026-02-13T02:26:30.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.453125 2026-02-13T02:26:30.493 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T02:26:30.494 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-13T02:26:30.494 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-13T02:26:30.494 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T02:26:30.494 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-snaps 1' TERM HUP INT 2026-02-13T02:26:30.494 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:39: run: TEST_scrub_snaps td/osd-scrub-snaps 2026-02-13T02:26:30.494 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:171: TEST_scrub_snaps: local dir=td/osd-scrub-snaps 2026-02-13T02:26:30.494 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:172: TEST_scrub_snaps: local poolname=test 2026-02-13T02:26:30.494 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:173: TEST_scrub_snaps: local OBJS=16 2026-02-13T02:26:30.494 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:174: TEST_scrub_snaps: local OSDS=1 2026-02-13T02:26:30.494 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:176: TEST_scrub_snaps: TESTDATA=testdata.453125 2026-02-13T02:26:30.494 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:178: TEST_scrub_snaps: run_mon td/osd-scrub-snaps a --osd_pool_default_size=1 2026-02-13T02:26:30.494 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-snaps 2026-02-13T02:26:30.494 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-13T02:26:30.494 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-13T02:26:30.494 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-13T02:26:30.494 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-snaps/a 2026-02-13T02:26:30.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-snaps/a --run-dir=td/osd-scrub-snaps --osd_pool_default_size=1 2026-02-13T02:26:30.515 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-13T02:26:30.515 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:26:30.515 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:26:30.515 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:26:30.515 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:26:30.516 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.453125 2026-02-13T02:26:30.516 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.453125/$cluster-$name.asok' 2026-02-13T02:26:30.516 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-snaps/a '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.453125/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-snaps/log --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 2026-02-13T02:26:30.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-13T02:26:30.545 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-13T02:26:30.545 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:26:30.545 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:26:30.545 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-13T02:26:30.545 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-13T02:26:30.545 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:26:30.545 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:26:30.545 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:26:30.546 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:26:30.546 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:26:30.546 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.453125 2026-02-13T02:26:30.546 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.453125/ceph-mon.a.asok 2026-02-13T02:26:30.546 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:26:30.546 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.453125/ceph-mon.a.asok config get fsid 2026-02-13T02:26:30.616 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-13T02:26:30.616 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:26:30.616 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:26:30.616 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-13T02:26:30.616 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-13T02:26:30.616 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:26:30.616 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:26:30.617 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:26:30.617 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:26:30.617 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:26:30.617 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.453125 2026-02-13T02:26:30.617 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.453125/ceph-mon.a.asok 2026-02-13T02:26:30.617 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:26:30.617 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.453125/ceph-mon.a.asok config get mon_host 2026-02-13T02:26:30.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:179: TEST_scrub_snaps: run_mgr td/osd-scrub-snaps x 2026-02-13T02:26:30.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-snaps 2026-02-13T02:26:30.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-13T02:26:30.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-13T02:26:30.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-13T02:26:30.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-snaps/x 2026-02-13T02:26:30.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-13T02:26:30.767 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-13T02:26:30.767 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:26:30.767 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:26:30.767 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:26:30.767 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:26:30.767 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.453125 2026-02-13T02:26:30.768 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.453125/$cluster-$name.asok' 2026-02-13T02:26:30.768 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:26:30.769 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-snaps/x '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.453125/$cluster-$name.asok' --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:26:30.786 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:180: TEST_scrub_snaps: expr 1 - 1 2026-02-13T02:26:30.787 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:180: TEST_scrub_snaps: seq 0 0 2026-02-13T02:26:30.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:180: TEST_scrub_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:26:30.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:182: TEST_scrub_snaps: run_osd td/osd-scrub-snaps 0 2026-02-13T02:26:30.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-snaps 2026-02-13T02:26:30.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:26:30.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-13T02:26:30.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:26:30.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-snaps/0 2026-02-13T02:26:30.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=b2b95964-f5a3-4b20-bd2b-d60f1ae5f8a0 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2026-02-13T02:26:30.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:26:30.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:26:30.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:26:30.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2026-02-13T02:26:30.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2026-02-13T02:26:30.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:26:30.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:26:30.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2026-02-13T02:26:30.789 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:26:30.789 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:26:30.789 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:26:30.789 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:26:30.789 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:26:30.789 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.453125 2026-02-13T02:26:30.789 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.453125/$cluster-$name.asok' 2026-02-13T02:26:30.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.453125/$cluster-$name.asok' 2026-02-13T02:26:30.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:26:30.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:26:30.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:26:30.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2026-02-13T02:26:30.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2026-02-13T02:26:30.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:26:30.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:26:30.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:26:30.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:26:30.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:26:30.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:26:30.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-snaps/0 2026-02-13T02:26:30.791 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:26:30.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d6ef9276-8ea4-4f26-979c-09dda6c2bb52 2026-02-13T02:26:30.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 d6ef9276-8ea4-4f26-979c-09dda6c2bb52' 2026-02-13T02:26:30.792 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 d6ef9276-8ea4-4f26-979c-09dda6c2bb52 2026-02-13T02:26:30.792 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:26:30.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDWi45puEt6LxAA5sVWTLBfyid2XrAO2rl/xQ== 2026-02-13T02:26:30.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDWi45puEt6LxAA5sVWTLBfyid2XrAO2rl/xQ=="}' 2026-02-13T02:26:30.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d6ef9276-8ea4-4f26-979c-09dda6c2bb52 -i td/osd-scrub-snaps/0/new.json 2026-02-13T02:26:30.894 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:26:30.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-snaps/0/new.json 2026-02-13T02:26:30.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=b2b95964-f5a3-4b20-bd2b-d60f1ae5f8a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.453125/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDWi45puEt6LxAA5sVWTLBfyid2XrAO2rl/xQ== --osd-uuid d6ef9276-8ea4-4f26-979c-09dda6c2bb52 2026-02-13T02:26:30.918 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:30.907+0000 7fd836c1ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:26:30.919 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:30.907+0000 7fd836c1ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:26:30.920 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:30.911+0000 7fd836c1ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:26:30.921 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:30.911+0000 7fd836c1ea40 -1 bdev(0x56183bdbcc00 td/osd-scrub-snaps/0/block) open stat got: (1) Operation not permitted 2026-02-13T02:26:30.921 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:30.911+0000 7fd836c1ea40 -1 bluestore(td/osd-scrub-snaps/0) _read_fsid unparsable uuid 2026-02-13T02:26:31.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-snaps/0/keyring 2026-02-13T02:26:31.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:26:31.632 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T02:26:31.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-13T02:26:31.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-snaps/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:26:31.733 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T02:26:31.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-13T02:26:31.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=b2b95964-f5a3-4b20-bd2b-d60f1ae5f8a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.453125/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:26:31.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:26:31.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:26:31.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:26:31.755 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:31.743+0000 7f4565ad1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:26:31.757 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:31.747+0000 7f4565ad1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:26:31.758 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:31.747+0000 7f4565ad1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:26:31.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-13T02:26:31.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:26:31.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:26:31.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:26:31.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:26:31.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:26:31.829 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:26:31.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:26:31.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:26:31.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:26:31.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:26:32.258 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:32.247+0000 7f4565ad1a40 -1 Falling back to public interface 2026-02-13T02:26:32.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:26:32.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:26:32.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:26:32.921 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:26:32.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:26:32.926 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:26:33.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:26:33.230 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:33.219+0000 7f4565ad1a40 -1 osd.0 0 log_to_monitors true 2026-02-13T02:26:34.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:26:34.047 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:26:34.047 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:26:34.048 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:26:34.048 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:26:34.048 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:26:34.211 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/175904387,v1:127.0.0.1:6801/175904387] [v2:127.0.0.1:6802/175904387,v1:127.0.0.1:6803/175904387] exists,up d6ef9276-8ea4-4f26-979c-09dda6c2bb52 2026-02-13T02:26:34.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:26:34.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:26:34.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:26:34.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:186: TEST_scrub_snaps: ceph osd set noscrub 2026-02-13T02:26:34.452 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:26:34.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:187: TEST_scrub_snaps: ceph osd set nodeep-scrub 2026-02-13T02:26:34.656 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:26:34.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:190: TEST_scrub_snaps: create_pool test 1 1 2026-02-13T02:26:34.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2026-02-13T02:26:34.859 INFO:tasks.workunit.client.0.trial112.stderr:pool 'test' created 2026-02-13T02:26:34.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:26:35.869 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:191: TEST_scrub_snaps: wait_for_clean 2026-02-13T02:26:35.870 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:26:35.870 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:26:35.870 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:26:35.870 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:26:35.871 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:26:35.871 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:26:35.871 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:26:35.871 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:26:35.871 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:26:35.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:26:35.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:26:35.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:26:35.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:26:35.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:26:35.965 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:26:36.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-13T02:26:36.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:26:36.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:26:36.140 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:26:36.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836482 2026-02-13T02:26:36.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836482 2026-02-13T02:26:36.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836482' 2026-02-13T02:26:36.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:26:36.215 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836482 2026-02-13T02:26:36.215 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:26:36.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:26:36.217 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836482 2026-02-13T02:26:36.217 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:26:36.218 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836482 2026-02-13T02:26:36.218 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836482' 2026-02-13T02:26:36.218 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836482 2026-02-13T02:26:36.218 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:26:36.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836481 -lt 21474836482 2026-02-13T02:26:36.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:26:37.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:26:37.389 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:26:37.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836482 2026-02-13T02:26:37.547 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:26:37.547 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:26:37.547 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:26:37.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:26:37.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:26:37.738 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:26:37.739 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:26:37.739 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:26:37.739 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:26:37.739 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:26:37.739 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:26:37.893 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:26:37.893 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:26:37.894 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:26:37.894 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:26:38.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:26:38.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:26:38.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:26:38.095 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: ceph osd dump 2026-02-13T02:26:38.095 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: grep '^pool.*['\'']test['\'']' 2026-02-13T02:26:38.095 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: awk '{ print $2 }' 2026-02-13T02:26:38.255 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: poolid=1 2026-02-13T02:26:38.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:194: TEST_scrub_snaps: dd if=/dev/urandom of=testdata.453125 bs=1032 count=1 2026-02-13T02:26:38.256 INFO:tasks.workunit.client.0.trial112.stderr:1+0 records in 2026-02-13T02:26:38.256 INFO:tasks.workunit.client.0.trial112.stderr:1+0 records out 2026-02-13T02:26:38.256 INFO:tasks.workunit.client.0.trial112.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 5.864e-05 s, 17.6 MB/s 2026-02-13T02:26:38.257 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: seq 1 16 2026-02-13T02:26:38.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-13T02:26:38.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj1 testdata.453125 2026-02-13T02:26:38.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-13T02:26:38.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj2 testdata.453125 2026-02-13T02:26:38.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-13T02:26:38.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj3 testdata.453125 2026-02-13T02:26:38.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-13T02:26:38.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj4 testdata.453125 2026-02-13T02:26:38.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-13T02:26:38.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj5 testdata.453125 2026-02-13T02:26:38.358 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-13T02:26:38.358 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj6 testdata.453125 2026-02-13T02:26:38.381 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-13T02:26:38.381 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj7 testdata.453125 2026-02-13T02:26:38.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-13T02:26:38.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj8 testdata.453125 2026-02-13T02:26:38.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-13T02:26:38.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj9 testdata.453125 2026-02-13T02:26:38.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-13T02:26:38.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj10 testdata.453125 2026-02-13T02:26:38.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-13T02:26:38.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj11 testdata.453125 2026-02-13T02:26:38.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-13T02:26:38.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj12 testdata.453125 2026-02-13T02:26:38.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-13T02:26:38.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj13 testdata.453125 2026-02-13T02:26:38.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-13T02:26:38.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj14 testdata.453125 2026-02-13T02:26:38.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-13T02:26:38.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj15 testdata.453125 2026-02-13T02:26:38.598 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-13T02:26:38.598 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj16 testdata.453125 2026-02-13T02:26:38.623 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:200: TEST_scrub_snaps: get_primary test obj1 2026-02-13T02:26:38.623 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2026-02-13T02:26:38.623 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-13T02:26:38.623 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2026-02-13T02:26:38.623 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T02:26:38.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:200: TEST_scrub_snaps: local primary=0 2026-02-13T02:26:38.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:202: TEST_scrub_snaps: create_scenario td/osd-scrub-snaps test testdata.453125 0 2026-02-13T02:26:38.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:45: create_scenario: local dir=td/osd-scrub-snaps 2026-02-13T02:26:38.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:46: create_scenario: local poolname=test 2026-02-13T02:26:38.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:47: create_scenario: local TESTDATA=testdata.453125 2026-02-13T02:26:38.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:48: create_scenario: local osd=0 2026-02-13T02:26:38.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:50: create_scenario: SNAP=1 2026-02-13T02:26:38.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:51: create_scenario: rados -p test mksnap snap1 2026-02-13T02:26:38.854 INFO:tasks.workunit.client.0.trial112.stdout:created pool test snap snap1 2026-02-13T02:26:38.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:52: create_scenario: dd if=/dev/urandom of=testdata.453125 bs=256 count=1 2026-02-13T02:26:38.857 INFO:tasks.workunit.client.0.trial112.stderr:1+0 records in 2026-02-13T02:26:38.857 INFO:tasks.workunit.client.0.trial112.stderr:1+0 records out 2026-02-13T02:26:38.857 INFO:tasks.workunit.client.0.trial112.stderr:256 bytes copied, 8.9988e-05 s, 2.8 MB/s 2026-02-13T02:26:38.857 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:53: create_scenario: rados -p test put obj1 testdata.453125 2026-02-13T02:26:38.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:54: create_scenario: rados -p test put obj5 testdata.453125 2026-02-13T02:26:38.905 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:55: create_scenario: rados -p test put obj3 testdata.453125 2026-02-13T02:26:38.930 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: seq 6 14 2026-02-13T02:26:38.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-13T02:26:38.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj6 testdata.453125 2026-02-13T02:26:38.950 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-13T02:26:38.950 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj7 testdata.453125 2026-02-13T02:26:38.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-13T02:26:38.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj8 testdata.453125 2026-02-13T02:26:38.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-13T02:26:38.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj9 testdata.453125 2026-02-13T02:26:39.021 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-13T02:26:39.021 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj10 testdata.453125 2026-02-13T02:26:39.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-13T02:26:39.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj11 testdata.453125 2026-02-13T02:26:39.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-13T02:26:39.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj12 testdata.453125 2026-02-13T02:26:39.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-13T02:26:39.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj13 testdata.453125 2026-02-13T02:26:39.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-13T02:26:39.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj14 testdata.453125 2026-02-13T02:26:39.144 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:60: create_scenario: SNAP=2 2026-02-13T02:26:39.144 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:61: create_scenario: rados -p test mksnap snap2 2026-02-13T02:26:39.217 INFO:tasks.workunit.client.0.trial112.stdout:created pool test snap snap2 2026-02-13T02:26:39.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:62: create_scenario: dd if=/dev/urandom of=testdata.453125 bs=256 count=2 2026-02-13T02:26:39.221 INFO:tasks.workunit.client.0.trial112.stderr:2+0 records in 2026-02-13T02:26:39.221 INFO:tasks.workunit.client.0.trial112.stderr:2+0 records out 2026-02-13T02:26:39.221 INFO:tasks.workunit.client.0.trial112.stderr:512 bytes copied, 8.4768e-05 s, 6.0 MB/s 2026-02-13T02:26:39.222 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:63: create_scenario: rados -p test put obj5 testdata.453125 2026-02-13T02:26:39.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:65: create_scenario: SNAP=3 2026-02-13T02:26:39.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:66: create_scenario: rados -p test mksnap snap3 2026-02-13T02:26:39.319 INFO:tasks.workunit.client.0.trial112.stdout:created pool test snap snap3 2026-02-13T02:26:39.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:67: create_scenario: dd if=/dev/urandom of=testdata.453125 bs=256 count=3 2026-02-13T02:26:39.323 INFO:tasks.workunit.client.0.trial112.stderr:3+0 records in 2026-02-13T02:26:39.324 INFO:tasks.workunit.client.0.trial112.stderr:3+0 records out 2026-02-13T02:26:39.324 INFO:tasks.workunit.client.0.trial112.stderr:768 bytes copied, 8.9186e-05 s, 8.6 MB/s 2026-02-13T02:26:39.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:68: create_scenario: rados -p test put obj3 testdata.453125 2026-02-13T02:26:39.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:70: create_scenario: SNAP=4 2026-02-13T02:26:39.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:71: create_scenario: rados -p test mksnap snap4 2026-02-13T02:26:39.420 INFO:tasks.workunit.client.0.trial112.stdout:created pool test snap snap4 2026-02-13T02:26:39.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:72: create_scenario: dd if=/dev/urandom of=testdata.453125 bs=256 count=4 2026-02-13T02:26:39.424 INFO:tasks.workunit.client.0.trial112.stderr:4+0 records in 2026-02-13T02:26:39.424 INFO:tasks.workunit.client.0.trial112.stderr:4+0 records out 2026-02-13T02:26:39.425 INFO:tasks.workunit.client.0.trial112.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 9.4817e-05 s, 10.8 MB/s 2026-02-13T02:26:39.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:73: create_scenario: rados -p test put obj5 testdata.453125 2026-02-13T02:26:39.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:74: create_scenario: rados -p test put obj2 testdata.453125 2026-02-13T02:26:39.474 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:76: create_scenario: SNAP=5 2026-02-13T02:26:39.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:77: create_scenario: rados -p test mksnap snap5 2026-02-13T02:26:39.524 INFO:tasks.workunit.client.0.trial112.stdout:created pool test snap snap5 2026-02-13T02:26:39.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:78: create_scenario: SNAP=6 2026-02-13T02:26:39.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:79: create_scenario: rados -p test mksnap snap6 2026-02-13T02:26:39.626 INFO:tasks.workunit.client.0.trial112.stdout:created pool test snap snap6 2026-02-13T02:26:39.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:80: create_scenario: dd if=/dev/urandom of=testdata.453125 bs=256 count=6 2026-02-13T02:26:39.630 INFO:tasks.workunit.client.0.trial112.stderr:6+0 records in 2026-02-13T02:26:39.630 INFO:tasks.workunit.client.0.trial112.stderr:6+0 records out 2026-02-13T02:26:39.630 INFO:tasks.workunit.client.0.trial112.stderr:1536 bytes (1.5 kB, 1.5 KiB) copied, 0.000103753 s, 14.8 MB/s 2026-02-13T02:26:39.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:81: create_scenario: rados -p test put obj5 testdata.453125 2026-02-13T02:26:39.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:83: create_scenario: SNAP=7 2026-02-13T02:26:39.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:84: create_scenario: rados -p test mksnap snap7 2026-02-13T02:26:39.727 INFO:tasks.workunit.client.0.trial112.stdout:created pool test snap snap7 2026-02-13T02:26:39.729 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:86: create_scenario: rados -p test rm obj4 2026-02-13T02:26:39.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:87: create_scenario: rados -p test rm obj16 2026-02-13T02:26:39.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:88: create_scenario: rados -p test rm obj2 2026-02-13T02:26:39.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:90: create_scenario: kill_daemons td/osd-scrub-snaps TERM osd 2026-02-13T02:26:39.789 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:26:39.789 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:26:39.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:26:39.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:26:39.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:26:40.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:26:40.098 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj1 2026-02-13T02:26:40.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: JSON='["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:26:40.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:95: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' --force remove 2026-02-13T02:26:40.488 INFO:tasks.workunit.client.0.trial112.stdout:WARNING: only removing head with clones present 2026-02-13T02:26:40.489 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:ff7b1f36:::obj1:head# 2026-02-13T02:26:40.585 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2026-02-13T02:26:40.585 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: grep '"snapid":2' 2026-02-13T02:26:40.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:26:40.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:98: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2026-02-13T02:26:40.903 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:c52c9666:::obj5:2# 2026-02-13T02:26:40.989 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2026-02-13T02:26:40.989 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: grep '"snapid":1' 2026-02-13T02:26:41.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:26:41.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:101: create_scenario: OBJ5SAVE='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:26:41.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:103: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2026-02-13T02:26:41.383 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:104: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2026-02-13T02:26:41.385 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:26:41.385 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2026-02-13T02:26:41.385 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:26:41.385 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-13T02:26:41.385 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2026-02-13T02:26:41.385 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2026-02-13T02:26:41.385 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2026-02-13T02:26:41.386 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2026-02-13T02:26:41.386 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2026-02-13T02:26:41.386 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2026-02-13T02:26:41.386 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:26:41.386 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2026-02-13T02:26:41.386 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2026-02-13T02:26:41.386 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2026-02-13T02:26:41.386 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2026-02-13T02:26:41.386 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:26:41.386 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2026-02-13T02:26:41.386 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:26:41.386 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:26:41.387 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:26:41.387 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2026-02-13T02:26:41.387 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:26:41.387 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2026-02-13T02:26:41.387 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:26:41.387 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:26:41.387 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:26:41.387 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2026-02-13T02:26:41.387 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:26:41.387 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:26:41.387 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:26:41.387 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2026-02-13T02:26:41.387 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2026-02-13T02:26:41.388 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:26:41.388 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:26:41.388 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2026-02-13T02:26:41.388 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2026-02-13T02:26:41.388 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:26:41.388 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:26:41.388 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2026-02-13T02:26:41.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:105: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2026-02-13T02:26:41.389 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-13T02:26:41.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:106: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype nosnapmap '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2026-02-13T02:26:41.564 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:c52c9666:::obj5:1# 2026-02-13T02:26:41.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:108: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2026-02-13T02:26:41.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:109: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2026-02-13T02:26:41.909 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:26:41.909 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2026-02-13T02:26:41.909 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:26:41.909 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-13T02:26:41.909 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2026-02-13T02:26:41.909 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2026-02-13T02:26:41.909 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2026-02-13T02:26:41.909 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2026-02-13T02:26:41.910 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2026-02-13T02:26:41.910 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2026-02-13T02:26:41.910 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:26:41.910 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2026-02-13T02:26:41.910 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2026-02-13T02:26:41.910 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2026-02-13T02:26:41.910 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2026-02-13T02:26:41.910 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:26:41.910 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2026-02-13T02:26:41.910 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:26:41.910 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:26:41.910 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:26:41.910 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2026-02-13T02:26:41.911 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:26:41.911 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2026-02-13T02:26:41.911 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:26:41.911 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:26:41.911 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:26:41.911 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2026-02-13T02:26:41.911 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:26:41.911 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:26:41.911 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:26:41.911 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2026-02-13T02:26:41.911 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2026-02-13T02:26:41.911 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:26:41.911 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:26:41.912 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2026-02-13T02:26:41.912 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2026-02-13T02:26:41.912 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:26:41.912 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:26:41.912 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2026-02-13T02:26:41.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:110: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2026-02-13T02:26:41.913 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-13T02:26:41.913 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:111: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2026-02-13T02:26:41.913 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2026-02-13T02:26:41.913 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: grep '"snapid":4' 2026-02-13T02:26:42.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:26:42.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:114: create_scenario: dd if=/dev/urandom of=testdata.453125 bs=256 count=18 2026-02-13T02:26:42.071 INFO:tasks.workunit.client.0.trial112.stderr:18+0 records in 2026-02-13T02:26:42.071 INFO:tasks.workunit.client.0.trial112.stderr:18+0 records out 2026-02-13T02:26:42.071 INFO:tasks.workunit.client.0.trial112.stderr:4608 bytes (4.6 kB, 4.5 KiB) copied, 0.000172352 s, 26.7 MB/s 2026-02-13T02:26:42.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:115: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.453125 2026-02-13T02:26:42.324 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj3 2026-02-13T02:26:42.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: JSON='["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:26:42.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:118: create_scenario: dd if=/dev/urandom of=testdata.453125 bs=256 count=15 2026-02-13T02:26:42.493 INFO:tasks.workunit.client.0.trial112.stderr:15+0 records in 2026-02-13T02:26:42.493 INFO:tasks.workunit.client.0.trial112.stderr:15+0 records out 2026-02-13T02:26:42.493 INFO:tasks.workunit.client.0.trial112.stderr:3840 bytes (3.8 kB, 3.8 KiB) copied, 9.617e-05 s, 39.9 MB/s 2026-02-13T02:26:42.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:119: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.453125 2026-02-13T02:26:42.789 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj4 2026-02-13T02:26:42.789 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: grep '"snapid":7' 2026-02-13T02:26:42.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: JSON='["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:26:42.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:122: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' remove 2026-02-13T02:26:43.120 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:0ee9ae15:::obj4:7# 2026-02-13T02:26:43.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:125: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2026-02-13T02:26:43.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:126: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2026-02-13T02:26:43.449 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2026-02-13T02:26:43.449 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:26:43.449 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-13T02:26:43.449 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2026-02-13T02:26:43.449 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2026-02-13T02:26:43.449 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2026-02-13T02:26:43.449 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2026-02-13T02:26:43.449 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2026-02-13T02:26:43.449 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2026-02-13T02:26:43.449 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:26:43.450 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2026-02-13T02:26:43.450 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2026-02-13T02:26:43.450 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2026-02-13T02:26:43.450 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2026-02-13T02:26:43.450 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2026-02-13T02:26:43.450 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:26:43.450 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:26:43.450 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2026-02-13T02:26:43.450 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:26:43.450 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2026-02-13T02:26:43.450 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:26:43.450 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:26:43.451 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2026-02-13T02:26:43.451 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:26:43.451 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:26:43.451 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2026-02-13T02:26:43.451 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2026-02-13T02:26:43.451 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:26:43.451 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2026-02-13T02:26:43.451 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2026-02-13T02:26:43.451 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:26:43.451 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2026-02-13T02:26:43.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:127: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2026-02-13T02:26:43.452 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:26:43.452 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:26:43.452 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:26:43.452 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:26:43.452 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:26:43.452 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:26:43.453 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:26:43.453 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj16 2026-02-13T02:26:43.453 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: grep '"snapid":7' 2026-02-13T02:26:43.620 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: JSON='["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:26:43.620 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:129: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype snapmap '["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' remove 2026-02-13T02:26:43.800 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:461f8b5e:::obj16:7# 2026-02-13T02:26:43.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:131: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2026-02-13T02:26:44.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:132: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2026-02-13T02:26:44.089 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2026-02-13T02:26:44.089 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-13T02:26:44.089 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2026-02-13T02:26:44.089 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2026-02-13T02:26:44.089 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2026-02-13T02:26:44.089 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2026-02-13T02:26:44.089 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2026-02-13T02:26:44.089 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2026-02-13T02:26:44.089 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:26:44.090 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2026-02-13T02:26:44.090 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2026-02-13T02:26:44.090 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2026-02-13T02:26:44.090 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2026-02-13T02:26:44.090 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2026-02-13T02:26:44.090 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:26:44.090 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2026-02-13T02:26:44.090 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2026-02-13T02:26:44.090 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:26:44.090 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2026-02-13T02:26:44.090 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:26:44.090 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2026-02-13T02:26:44.090 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2026-02-13T02:26:44.091 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2026-02-13T02:26:44.091 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2026-02-13T02:26:44.091 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2026-02-13T02:26:44.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:133: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2026-02-13T02:26:44.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:134: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2026-02-13T02:26:44.091 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj2 2026-02-13T02:26:44.280 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: JSON='["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:26:44.280 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:137: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' rm-attr snapset 2026-02-13T02:26:44.569 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: echo '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:26:44.570 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: sed 's/snapid":1/snapid":7/' 2026-02-13T02:26:44.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:26:44.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:141: create_scenario: dd if=/dev/urandom of=testdata.453125 bs=256 count=7 2026-02-13T02:26:44.572 INFO:tasks.workunit.client.0.trial112.stderr:7+0 records in 2026-02-13T02:26:44.572 INFO:tasks.workunit.client.0.trial112.stderr:7+0 records out 2026-02-13T02:26:44.572 INFO:tasks.workunit.client.0.trial112.stderr:1792 bytes (1.8 kB, 1.8 KiB) copied, 9.115e-05 s, 19.7 MB/s 2026-02-13T02:26:44.572 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:142: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.453125 2026-02-13T02:26:44.841 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj6 2026-02-13T02:26:44.907 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-13T02:26:44.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: JSON='["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:26:44.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:145: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset 2026-02-13T02:26:45.246 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj7 2026-02-13T02:26:45.327 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-13T02:26:45.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: JSON='["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:26:45.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:147: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset corrupt 2026-02-13T02:26:45.677 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj8 2026-02-13T02:26:45.755 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-13T02:26:45.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: JSON='["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:26:45.852 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:149: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset seq 2026-02-13T02:26:46.132 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj9 2026-02-13T02:26:46.209 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-13T02:26:46.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: JSON='["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:26:46.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:151: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_size 2026-02-13T02:26:46.617 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj10 2026-02-13T02:26:46.683 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-13T02:26:46.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: JSON='["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:26:46.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:153: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_overlap 2026-02-13T02:26:47.081 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj11 2026-02-13T02:26:47.163 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-13T02:26:47.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: JSON='["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:26:47.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:155: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clones 2026-02-13T02:26:47.537 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj12 2026-02-13T02:26:47.619 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-13T02:26:47.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: JSON='["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:26:47.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:157: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset head 2026-02-13T02:26:47.977 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj13 2026-02-13T02:26:48.059 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-13T02:26:48.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: JSON='["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:26:48.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:159: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset snaps 2026-02-13T02:26:48.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj14 2026-02-13T02:26:48.507 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-13T02:26:48.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: JSON='["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:26:48.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:161: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset size 2026-02-13T02:26:48.892 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:163: create_scenario: echo garbage 2026-02-13T02:26:48.893 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj15 2026-02-13T02:26:48.963 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-13T02:26:49.064 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: JSON='["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:26:49.064 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:165: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' set-attr snapset td/osd-scrub-snaps/bad 2026-02-13T02:26:49.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:166: create_scenario: rm -f td/osd-scrub-snaps/bad 2026-02-13T02:26:49.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:167: create_scenario: return 0 2026-02-13T02:26:49.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:204: TEST_scrub_snaps: rm -f testdata.453125 2026-02-13T02:26:49.327 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:206: TEST_scrub_snaps: expr 1 - 1 2026-02-13T02:26:49.329 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:206: TEST_scrub_snaps: seq 0 0 2026-02-13T02:26:49.331 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:206: TEST_scrub_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:26:49.331 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:208: TEST_scrub_snaps: activate_osd td/osd-scrub-snaps 0 2026-02-13T02:26:49.331 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-snaps 2026-02-13T02:26:49.332 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:26:49.332 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:26:49.332 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:26:49.332 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-snaps/0 2026-02-13T02:26:49.332 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=b2b95964-f5a3-4b20-bd2b-d60f1ae5f8a0 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2026-02-13T02:26:49.332 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:26:49.332 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:26:49.332 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:26:49.332 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2026-02-13T02:26:49.332 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2026-02-13T02:26:49.332 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:26:49.332 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:26:49.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2026-02-13T02:26:49.333 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:26:49.333 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:26:49.333 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:26:49.333 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:26:49.333 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:26:49.333 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.453125 2026-02-13T02:26:49.333 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.453125/$cluster-$name.asok' 2026-02-13T02:26:49.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.453125/$cluster-$name.asok' 2026-02-13T02:26:49.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:26:49.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2026-02-13T02:26:49.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2026-02-13T02:26:49.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:26:49.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:26:49.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:26:49.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:26:49.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:26:49.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:26:49.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-snaps/0 2026-02-13T02:26:49.335 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T02:26:49.336 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:26:49.336 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-snaps/0/whoami 2026-02-13T02:26:49.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=b2b95964-f5a3-4b20-bd2b-d60f1ae5f8a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.453125/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:26:49.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:26:49.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:26:49.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:26:49.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:26:49.353 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:49.343+0000 7f36b579da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:26:49.354 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:49.343+0000 7f36b579da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:26:49.355 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:49.343+0000 7f36b579da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:26:49.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:26:49.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:26:49.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:26:49.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:26:49.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:26:49.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:26:49.496 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:26:49.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:26:49.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:26:49.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:26:49.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:26:49.770 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:49.759+0000 7f36b579da40 -1 Falling back to public interface 2026-02-13T02:26:50.314 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:26:50.303+0000 7f36b579da40 -1 osd.0 18 log_to_monitors true 2026-02-13T02:26:50.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:26:50.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:26:50.651 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:26:50.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:26:50.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:26:50.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:26:50.818 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 21 up_thru 21 down_at 19 last_clean_interval [5,18) [v2:127.0.0.1:6800/3886909364,v1:127.0.0.1:6801/3886909364] [v2:127.0.0.1:6802/3886909364,v1:127.0.0.1:6803/3886909364] exists,up d6ef9276-8ea4-4f26-979c-09dda6c2bb52 2026-02-13T02:26:50.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:26:50.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:26:50.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:26:50.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:210: TEST_scrub_snaps: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_max 25 2026-02-13T02:26:50.883 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:26:50.883 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-13T02:26:50.883 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:26:50.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:211: TEST_scrub_snaps: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_min 5 2026-02-13T02:26:50.968 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:26:50.968 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_shallow_scrub_chunk_min = '' " 2026-02-13T02:26:50.968 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:26:50.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:212: TEST_scrub_snaps: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_seconds 1 2026-02-13T02:26:51.047 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:26:51.047 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_pg_stat_report_interval_max_seconds = '' " 2026-02-13T02:26:51.047 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:26:51.054 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:213: TEST_scrub_snaps: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_epochs 1 2026-02-13T02:26:51.126 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:26:51.126 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_pg_stat_report_interval_max_epochs = '' " 2026-02-13T02:26:51.127 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:26:51.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:216: TEST_scrub_snaps: wait_for_clean 2026-02-13T02:26:51.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:26:51.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:26:51.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:26:51.134 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:26:51.135 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:26:51.135 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:26:51.135 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:26:51.135 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:26:51.136 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:26:51.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:26:51.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:26:51.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:26:51.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:26:51.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:26:51.250 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:26:51.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-13T02:26:51.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:26:51.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:26:51.409 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:26:51.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=90194313218 2026-02-13T02:26:51.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 90194313218 2026-02-13T02:26:51.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-90194313218' 2026-02-13T02:26:51.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:26:51.483 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-90194313218 2026-02-13T02:26:51.483 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:26:51.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:26:51.485 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-90194313218 2026-02-13T02:26:51.485 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:26:51.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=90194313218 2026-02-13T02:26:51.486 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 90194313218 2026-02-13T02:26:51.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 90194313218' 2026-02-13T02:26:51.487 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:26:51.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 90194313218 2026-02-13T02:26:51.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:26:52.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:26:52.654 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:26:52.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 90194313218 2026-02-13T02:26:52.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:26:53.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:26:53.822 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:26:53.981 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 90194313218 -lt 90194313218 2026-02-13T02:26:53.981 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:26:53.982 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:26:53.982 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:26:54.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:26:54.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:26:54.174 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:26:54.174 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:26:54.174 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:26:54.174 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:26:54.174 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:26:54.174 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:26:54.331 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:26:54.332 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:26:54.332 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:26:54.332 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:26:54.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:26:54.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:26:54.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:26:54.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:218: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_shallow_scrub_chunk_max 2026-02-13T02:26:54.592 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:26:54.593 INFO:tasks.workunit.client.0.trial112.stdout: "osd_shallow_scrub_chunk_max": "25" 2026-02-13T02:26:54.593 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:26:54.600 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:219: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_shallow_scrub_chunk_min 2026-02-13T02:26:54.665 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:26:54.665 INFO:tasks.workunit.client.0.trial112.stdout: "osd_shallow_scrub_chunk_min": "5" 2026-02-13T02:26:54.665 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:26:54.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:220: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_pg_stat_report_interval_max_seconds 2026-02-13T02:26:54.739 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:26:54.739 INFO:tasks.workunit.client.0.trial112.stdout: "osd_pg_stat_report_interval_max_seconds": "1" 2026-02-13T02:26:54.739 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:26:54.747 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:221: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_pg_stat_report_interval_max_epochs 2026-02-13T02:26:54.821 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:26:54.822 INFO:tasks.workunit.client.0.trial112.stdout: "osd_pg_stat_report_interval_max_epochs": "1" 2026-02-13T02:26:54.822 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:26:54.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:222: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_scrub_chunk_max 2026-02-13T02:26:54.899 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:26:54.899 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_chunk_max": "15" 2026-02-13T02:26:54.899 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:26:54.906 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:223: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_scrub_chunk_min 2026-02-13T02:26:54.978 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:26:54.978 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_chunk_min": "5" 2026-02-13T02:26:54.978 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:26:54.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:225: TEST_scrub_snaps: local pgid=1.0 2026-02-13T02:26:54.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:226: TEST_scrub_snaps: pg_scrub 1.0 2026-02-13T02:26:54.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=1.0 2026-02-13T02:26:54.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 1.0 2026-02-13T02:26:54.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=1.0 2026-02-13T02:26:54.986 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-13T02:26:54.987 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:26:54.987 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:26:54.987 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:26:54.987 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:26:54.987 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:26:55.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-13T02:26:55.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-13T02:26:55.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-13T02:26:55.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-13T02:26:55.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:26:55.249 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:26:55.414 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-13T02:26:55.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:26:55.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:26:55.415 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:26:55.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=90194313220 2026-02-13T02:26:55.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 90194313220 2026-02-13T02:26:55.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-90194313220' 2026-02-13T02:26:55.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:26:55.486 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-90194313220 2026-02-13T02:26:55.486 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:26:55.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:26:55.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-90194313220 2026-02-13T02:26:55.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:26:55.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=90194313220 2026-02-13T02:26:55.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 90194313220' 2026-02-13T02:26:55.490 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 90194313220 2026-02-13T02:26:55.490 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:26:55.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 90194313218 -lt 90194313220 2026-02-13T02:26:55.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:26:56.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:26:56.653 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:26:56.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 90194313218 -lt 90194313220 2026-02-13T02:26:56.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:26:57.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:26:57.819 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:26:57.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 90194313220 -lt 90194313220 2026-02-13T02:26:57.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-13T02:26:57.979 INFO:tasks.workunit.client.0.trial112.stdout:#---------- 1.0 loop 0 2026-02-13T02:26:57.980 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 0' 2026-02-13T02:26:57.980 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-13T02:26:57.980 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-13T02:26:57.980 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-13T02:26:57.980 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-13T02:26:57.981 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-13T02:26:58.048 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-13T02:26:58.048 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-13T02:26:58.049 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-13T02:26:58.049 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-13T02:26:58.049 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: get_last_scrub_stamp 1.0 2026-02-13T02:26:58.050 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:26:58.050 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:26:58.050 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:26:58.050 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:26:58.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: local last_scrub=2026-02-13T02:26:34.850729+0000 2026-02-13T02:26:58.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_scrub: ceph pg scrub 1.0 2026-02-13T02:26:58.355 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 1.0 on osd.0 to scrub 2026-02-13T02:26:58.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_scrub: wait_for_scrub 1.0 2026-02-13T02:26:34.850729+0000 2026-02-13T02:26:58.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-13T02:26:58.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:26:34.850729+0000 2026-02-13T02:26:58.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:26:58.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:26:58.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:26:58.365 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:26:58.365 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:26:58.366 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:26:58.366 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:26:58.366 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:26:58.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:26:34.850729+0000 '>' 2026-02-13T02:26:34.850729+0000 2026-02-13T02:26:58.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:26:59.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:26:59.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:26:59.526 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:26:59.526 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:26:59.526 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:26:59.527 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:26:59.527 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:26:59.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:26:34.850729+0000 '>' 2026-02-13T02:26:34.850729+0000 2026-02-13T02:26:59.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:27:00.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:27:00.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:27:00.686 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:27:00.686 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:27:00.686 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:27:00.687 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:27:00.687 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:27:00.848 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:26:34.850729+0000 '>' 2026-02-13T02:26:34.850729+0000 2026-02-13T02:27:00.848 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:27:01.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:27:01.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:27:01.850 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:27:01.851 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:27:01.851 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:27:01.851 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:27:01.851 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:27:02.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:26:59.233485+0000 '>' 2026-02-13T02:26:34.850729+0000 2026-02-13T02:27:02.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:27:02.014 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:230: TEST_scrub_snaps: grep '_scan_snaps start' td/osd-scrub-snaps/osd.0.log 2026-02-13T02:27:02.014 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:230: TEST_scrub_snaps: wc -l 2026-02-13T02:27:02.017 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:230: TEST_scrub_snaps: test 2 = 2 2026-02-13T02:27:02.017 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:232: TEST_scrub_snaps: rados list-inconsistent-pg test 2026-02-13T02:27:02.041 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:234: TEST_scrub_snaps: jq '. | length' td/osd-scrub-snaps/json 2026-02-13T02:27:02.055 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:234: TEST_scrub_snaps: test 1 = 1 2026-02-13T02:27:02.056 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:236: TEST_scrub_snaps: jq -r '.[0]' td/osd-scrub-snaps/json 2026-02-13T02:27:02.065 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:236: TEST_scrub_snaps: test 1.0 = 1.0 2026-02-13T02:27:02.065 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:238: TEST_scrub_snaps: rados list-inconsistent-obj 1.0 2026-02-13T02:27:02.082 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:243: TEST_scrub_snaps: jq .inconsistents 2026-02-13T02:27:02.082 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:243: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2026-02-13T02:27:02.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:250: TEST_scrub_snaps: jq .inconsistents td/osd-scrub-snaps/json 2026-02-13T02:27:02.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:250: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2026-02-13T02:27:02.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:251: TEST_scrub_snaps: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2026-02-13T02:27:02.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2513: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2026-02-13T02:27:02.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:253: TEST_scrub_snaps: rados list-inconsistent-snapset 1.0 2026-02-13T02:27:02.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:255: TEST_scrub_snaps: jq .inconsistents 2026-02-13T02:27:02.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:255: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2026-02-13T02:27:02.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:634: TEST_scrub_snaps: jq .inconsistents td/osd-scrub-snaps/json 2026-02-13T02:27:02.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:634: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2026-02-13T02:27:02.167 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:635: TEST_scrub_snaps: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2026-02-13T02:27:02.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2513: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2026-02-13T02:27:02.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:636: TEST_scrub_snaps: test no = yes 2026-02-13T02:27:02.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:641: TEST_scrub_snaps: test '' = yes 2026-02-13T02:27:02.169 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:646: TEST_scrub_snaps: find td/osd-scrub-snaps 2026-02-13T02:27:02.169 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:646: TEST_scrub_snaps: grep 'osd[^/]*\.pid' 2026-02-13T02:27:02.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:646: TEST_scrub_snaps: pidfiles=td/osd-scrub-snaps/osd.0.pid 2026-02-13T02:27:02.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:647: TEST_scrub_snaps: pids= 2026-02-13T02:27:02.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:648: TEST_scrub_snaps: for pidfile in ${pidfiles} 2026-02-13T02:27:02.172 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:650: TEST_scrub_snaps: cat td/osd-scrub-snaps/osd.0.pid 2026-02-13T02:27:02.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:650: TEST_scrub_snaps: pids+='468429 ' 2026-02-13T02:27:02.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:653: TEST_scrub_snaps: ERRORS=0 2026-02-13T02:27:02.173 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: seq 1 7 2026-02-13T02:27:02.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2026-02-13T02:27:02.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap1 2026-02-13T02:27:02.241 INFO:tasks.workunit.client.0.trial112.stdout:removed pool test snap snap1 2026-02-13T02:27:02.243 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2026-02-13T02:27:02.243 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap2 2026-02-13T02:27:02.343 INFO:tasks.workunit.client.0.trial112.stdout:removed pool test snap snap2 2026-02-13T02:27:02.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2026-02-13T02:27:02.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap3 2026-02-13T02:27:02.445 INFO:tasks.workunit.client.0.trial112.stdout:removed pool test snap snap3 2026-02-13T02:27:02.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2026-02-13T02:27:02.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap4 2026-02-13T02:27:02.547 INFO:tasks.workunit.client.0.trial112.stdout:removed pool test snap snap4 2026-02-13T02:27:02.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2026-02-13T02:27:02.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap5 2026-02-13T02:27:02.649 INFO:tasks.workunit.client.0.trial112.stdout:removed pool test snap snap5 2026-02-13T02:27:02.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2026-02-13T02:27:02.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap6 2026-02-13T02:27:02.751 INFO:tasks.workunit.client.0.trial112.stdout:removed pool test snap snap6 2026-02-13T02:27:02.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2026-02-13T02:27:02.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap7 2026-02-13T02:27:02.853 INFO:tasks.workunit.client.0.trial112.stdout:removed pool test snap snap7 2026-02-13T02:27:02.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:659: TEST_scrub_snaps: sleep 5 2026-02-13T02:27:07.857 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:660: TEST_scrub_snaps: local -i loop=0 2026-02-13T02:27:07.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:661: TEST_scrub_snaps: ceph pg dump pgs 2026-02-13T02:27:07.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:661: TEST_scrub_snaps: grep -q snaptrim 2026-02-13T02:27:08.000 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:27:08.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:663: TEST_scrub_snaps: ceph pg dump pgs 2026-02-13T02:27:08.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:663: TEST_scrub_snaps: grep -q snaptrim_error 2026-02-13T02:27:08.170 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:27:08.181 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:665: TEST_scrub_snaps: break 2026-02-13T02:27:08.181 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:674: TEST_scrub_snaps: ceph pg dump pgs 2026-02-13T02:27:08.340 INFO:tasks.workunit.client.0.trial112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-13T02:27:08.340 INFO:tasks.workunit.client.0.trial112.stdout:1.0 35 0 0 0 0 23416 0 0 60 0 60 active+clean+inconsistent+snaptrim_error 2026-02-13T02:27:02.847426+0000 23'60 29:139 [0] 0 [0] 0 18'56 2026-02-13T02:26:59.233485+0000 0'0 2026-02-13T02:26:34.850729+0000 7 0 periodic scrub scheduled @ 2026-02-14T12:05:01.401348+0000 33 0 2026-02-13T02:27:08.340 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T02:27:08.340 INFO:tasks.workunit.client.0.trial112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-13T02:27:08.341 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:27:08.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:676: TEST_scrub_snaps: for pid in $pids 2026-02-13T02:27:08.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:678: TEST_scrub_snaps: kill -0 468429 2026-02-13T02:27:08.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:685: TEST_scrub_snaps: kill_daemons td/osd-scrub-snaps 2026-02-13T02:27:08.350 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:27:08.350 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:27:08.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:27:08.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:27:08.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:27:13.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:27:13.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:687: TEST_scrub_snaps: declare -a err_strings 2026-02-13T02:27:13.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:688: TEST_scrub_snaps: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj10:.* : is missing in clone_overlap' 2026-02-13T02:27:13.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:689: TEST_scrub_snaps: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : no '\''_'\'' attr' 2026-02-13T02:27:13.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:690: TEST_scrub_snaps: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : is an unexpected clone' 2026-02-13T02:27:13.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:691: TEST_scrub_snaps: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:4 : on disk size [(]4608[)] does not match object info size [(]512[)] adjusted for ondisk to [(]512[)]' 2026-02-13T02:27:13.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:692: TEST_scrub_snaps: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:2' 2026-02-13T02:27:13.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:693: TEST_scrub_snaps: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:1' 2026-02-13T02:27:13.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:694: TEST_scrub_snaps: err_strings[6]='log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj5:head : 2 missing clone[(]s[)]' 2026-02-13T02:27:13.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:695: TEST_scrub_snaps: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj8:head : snaps.seq not set' 2026-02-13T02:27:13.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:696: TEST_scrub_snaps: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj7:1 : is an unexpected clone' 2026-02-13T02:27:13.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:697: TEST_scrub_snaps: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj3:head : on disk size [(]3840[)] does not match object info size [(]768[)] adjusted for ondisk to [(]768[)]' 2026-02-13T02:27:13.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:698: TEST_scrub_snaps: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj6:1 : is an unexpected clone' 2026-02-13T02:27:13.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:699: TEST_scrub_snaps: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:head : no '\''snapset'\'' attr' 2026-02-13T02:27:13.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:7 : clone ignored due to missing snapset' 2026-02-13T02:27:13.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:701: TEST_scrub_snaps: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:4 : clone ignored due to missing snapset' 2026-02-13T02:27:13.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj4:head : expected clone .*:::obj4:7' 2026-02-13T02:27:13.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:703: TEST_scrub_snaps: err_strings[15]='log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj4:head : 1 missing clone[(]s[)]' 2026-02-13T02:27:13.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:704: TEST_scrub_snaps: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj1:1 : is an unexpected clone' 2026-02-13T02:27:13.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:705: TEST_scrub_snaps: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj9:1 : is missing in clone_size' 2026-02-13T02:27:13.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:706: TEST_scrub_snaps: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj11:1 : is an unexpected clone' 2026-02-13T02:27:13.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:707: TEST_scrub_snaps: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj14:1 : size 1032 != clone_size 1033' 2026-02-13T02:27:13.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:708: TEST_scrub_snaps: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' 2026-02-13T02:27:13.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:709: TEST_scrub_snaps: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj15:head : can'\''t decode '\''snapset'\'' attr ' 2026-02-13T02:27:13.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: err_strings[22]='log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' 2026-02-13T02:27:13.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-13T02:27:13.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj10:.* : is missing in clone_overlap' td/osd-scrub-snaps/osd.0.log 2026-02-13T02:27:13.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-13T02:27:13.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : no '\''_'\'' attr' td/osd-scrub-snaps/osd.0.log 2026-02-13T02:27:13.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-13T02:27:13.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2026-02-13T02:27:13.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-13T02:27:13.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:4 : on disk size [(]4608[)] does not match object info size [(]512[)] adjusted for ondisk to [(]512[)]' td/osd-scrub-snaps/osd.0.log 2026-02-13T02:27:13.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-13T02:27:13.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:2' td/osd-scrub-snaps/osd.0.log 2026-02-13T02:27:13.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-13T02:27:13.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:1' td/osd-scrub-snaps/osd.0.log 2026-02-13T02:27:13.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-13T02:27:13.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj5:head : 2 missing clone[(]s[)]' td/osd-scrub-snaps/osd.0.log 2026-02-13T02:27:13.681 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-13T02:27:13.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj8:head : snaps.seq not set' td/osd-scrub-snaps/osd.0.log 2026-02-13T02:27:13.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-13T02:27:13.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj7:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2026-02-13T02:27:13.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-13T02:27:13.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj3:head : on disk size [(]3840[)] does not match object info size [(]768[)] adjusted for ondisk to [(]768[)]' td/osd-scrub-snaps/osd.0.log 2026-02-13T02:27:13.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-13T02:27:13.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj6:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2026-02-13T02:27:13.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-13T02:27:13.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:head : no '\''snapset'\'' attr' td/osd-scrub-snaps/osd.0.log 2026-02-13T02:27:13.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-13T02:27:13.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:7 : clone ignored due to missing snapset' td/osd-scrub-snaps/osd.0.log 2026-02-13T02:27:13.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-13T02:27:13.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:4 : clone ignored due to missing snapset' td/osd-scrub-snaps/osd.0.log 2026-02-13T02:27:13.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-13T02:27:13.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj4:head : expected clone .*:::obj4:7' td/osd-scrub-snaps/osd.0.log 2026-02-13T02:27:13.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-13T02:27:13.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj4:head : 1 missing clone[(]s[)]' td/osd-scrub-snaps/osd.0.log 2026-02-13T02:27:13.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-13T02:27:13.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj1:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2026-02-13T02:27:13.705 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-13T02:27:13.705 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj9:1 : is missing in clone_size' td/osd-scrub-snaps/osd.0.log 2026-02-13T02:27:13.707 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-13T02:27:13.707 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj11:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2026-02-13T02:27:13.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-13T02:27:13.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj14:1 : size 1032 != clone_size 1033' td/osd-scrub-snaps/osd.0.log 2026-02-13T02:27:13.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-13T02:27:13.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' td/osd-scrub-snaps/osd.0.log 2026-02-13T02:27:13.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-13T02:27:13.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj15:head : can'\''t decode '\''snapset'\'' attr ' td/osd-scrub-snaps/osd.0.log 2026-02-13T02:27:13.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-13T02:27:13.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' td/osd-scrub-snaps/osd.0.log 2026-02-13T02:27:13.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:721: TEST_scrub_snaps: '[' 0 '!=' 0 ']' 2026-02-13T02:27:13.719 INFO:tasks.workunit.client.0.trial112.stdout:TEST PASSED 2026-02-13T02:27:13.720 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:727: TEST_scrub_snaps: echo 'TEST PASSED' 2026-02-13T02:27:13.720 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:728: TEST_scrub_snaps: return 0 2026-02-13T02:27:13.720 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:40: run: teardown td/osd-scrub-snaps 2026-02-13T02:27:13.720 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2026-02-13T02:27:13.720 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:27:13.720 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2026-02-13T02:27:13.721 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:27:13.721 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:27:13.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:27:13.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:27:13.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:27:13.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:27:13.726 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:27:13.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:27:13.728 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:27:13.729 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:27:13.729 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:27:13.730 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:27:13.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:27:13.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:27:13.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:27:13.731 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:27:13.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:27:13.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:27:13.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2026-02-13T02:27:13.742 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:27:13.742 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:27:13.742 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.453125 2026-02-13T02:27:13.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.453125 2026-02-13T02:27:13.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:27:13.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:27:13.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:37: run: for func in $funcs 2026-02-13T02:27:13.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:38: run: setup td/osd-scrub-snaps 2026-02-13T02:27:13.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-snaps 2026-02-13T02:27:13.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-snaps 2026-02-13T02:27:13.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2026-02-13T02:27:13.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:27:13.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2026-02-13T02:27:13.745 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:27:13.745 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:27:13.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:27:13.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:27:13.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:27:13.747 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:27:13.747 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:27:13.748 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:27:13.748 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:27:13.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:27:13.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:27:13.749 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:27:13.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:27:13.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:27:13.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:27:13.751 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:27:13.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:27:13.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:27:13.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2026-02-13T02:27:13.755 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:27:13.755 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:27:13.755 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.453125 2026-02-13T02:27:13.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.453125 2026-02-13T02:27:13.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:27:13.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:27:13.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-snaps 2026-02-13T02:27:13.758 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T02:27:13.758 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:27:13.758 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.453125 2026-02-13T02:27:13.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.453125 2026-02-13T02:27:13.760 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T02:27:13.762 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-13T02:27:13.762 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T02:27:13.762 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-snaps 1' TERM HUP INT 2026-02-13T02:27:13.762 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:39: run: TEST_scrub_snaps_primary td/osd-scrub-snaps 2026-02-13T02:27:13.762 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1177: TEST_scrub_snaps_primary: local dir=td/osd-scrub-snaps 2026-02-13T02:27:13.762 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1178: TEST_scrub_snaps_primary: ORIG_ARGS='--fsid=b2b95964-f5a3-4b20-bd2b-d60f1ae5f8a0 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2026-02-13T02:27:13.762 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1179: TEST_scrub_snaps_primary: CEPH_ARGS+=' --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2026-02-13T02:27:13.762 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1180: TEST_scrub_snaps_primary: _scrub_snaps_multi td/osd-scrub-snaps primary 2026-02-13T02:27:13.762 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:732: _scrub_snaps_multi: local dir=td/osd-scrub-snaps 2026-02-13T02:27:13.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:733: _scrub_snaps_multi: local poolname=test 2026-02-13T02:27:13.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:734: _scrub_snaps_multi: local OBJS=16 2026-02-13T02:27:13.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:735: _scrub_snaps_multi: local OSDS=2 2026-02-13T02:27:13.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:736: _scrub_snaps_multi: local which=primary 2026-02-13T02:27:13.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:738: _scrub_snaps_multi: TESTDATA=testdata.453125 2026-02-13T02:27:13.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:740: _scrub_snaps_multi: run_mon td/osd-scrub-snaps a --osd_pool_default_size=2 2026-02-13T02:27:13.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-snaps 2026-02-13T02:27:13.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-13T02:27:13.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-13T02:27:13.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-13T02:27:13.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-snaps/a 2026-02-13T02:27:13.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-snaps/a --run-dir=td/osd-scrub-snaps --osd_pool_default_size=2 2026-02-13T02:27:13.786 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-13T02:27:13.786 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:27:13.786 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:27:13.787 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:27:13.787 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:27:13.787 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.453125 2026-02-13T02:27:13.787 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.453125/$cluster-$name.asok' 2026-02-13T02:27:13.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-snaps/a '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.453125/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-snaps/log --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2026-02-13T02:27:13.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-13T02:27:13.818 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-13T02:27:13.818 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:27:13.818 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:27:13.818 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-13T02:27:13.819 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-13T02:27:13.819 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:27:13.819 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:27:13.819 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:27:13.819 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:27:13.819 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:27:13.820 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.453125 2026-02-13T02:27:13.820 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.453125/ceph-mon.a.asok 2026-02-13T02:27:13.820 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:27:13.820 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.453125/ceph-mon.a.asok config get fsid 2026-02-13T02:27:13.876 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-13T02:27:13.876 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:27:13.876 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:27:13.876 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-13T02:27:13.878 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-13T02:27:13.878 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:27:13.878 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:27:13.878 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:27:13.878 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:27:13.878 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:27:13.878 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.453125 2026-02-13T02:27:13.878 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.453125/ceph-mon.a.asok 2026-02-13T02:27:13.879 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:27:13.879 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.453125/ceph-mon.a.asok config get mon_host 2026-02-13T02:27:13.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:741: _scrub_snaps_multi: run_mgr td/osd-scrub-snaps x 2026-02-13T02:27:13.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-snaps 2026-02-13T02:27:13.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-13T02:27:13.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-13T02:27:13.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-13T02:27:13.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-snaps/x 2026-02-13T02:27:13.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-13T02:27:14.033 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-13T02:27:14.033 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:27:14.033 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:27:14.034 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:27:14.034 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:27:14.034 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.453125 2026-02-13T02:27:14.034 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.453125/$cluster-$name.asok' 2026-02-13T02:27:14.035 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:27:14.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-snaps/x '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.453125/$cluster-$name.asok' --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:27:14.057 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: expr 2 - 1 2026-02-13T02:27:14.058 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: seq 0 1 2026-02-13T02:27:14.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:27:14.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:744: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 0 2026-02-13T02:27:14.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-snaps 2026-02-13T02:27:14.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:27:14.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-13T02:27:14.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:27:14.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-snaps/0 2026-02-13T02:27:14.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=b2b95964-f5a3-4b20-bd2b-d60f1ae5f8a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2026-02-13T02:27:14.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:27:14.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:27:14.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:27:14.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2026-02-13T02:27:14.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2026-02-13T02:27:14.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:27:14.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:27:14.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2026-02-13T02:27:14.063 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:27:14.063 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:27:14.064 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:27:14.064 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:27:14.064 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:27:14.064 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.453125 2026-02-13T02:27:14.064 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.453125/$cluster-$name.asok' 2026-02-13T02:27:14.064 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.453125/$cluster-$name.asok' 2026-02-13T02:27:14.064 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:27:14.064 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:27:14.064 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:27:14.064 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2026-02-13T02:27:14.064 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2026-02-13T02:27:14.064 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:27:14.064 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:27:14.065 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:27:14.065 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:27:14.065 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:27:14.065 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:27:14.065 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-snaps/0 2026-02-13T02:27:14.065 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:27:14.066 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b9e3d0f1-151c-4747-8a3c-057ae2476fe7 2026-02-13T02:27:14.066 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 b9e3d0f1-151c-4747-8a3c-057ae2476fe7' 2026-02-13T02:27:14.066 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 b9e3d0f1-151c-4747-8a3c-057ae2476fe7 2026-02-13T02:27:14.066 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:27:14.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQACjI5pKThNBBAArDAtEO8RYuAHnhq8NewdrA== 2026-02-13T02:27:14.079 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQACjI5pKThNBBAArDAtEO8RYuAHnhq8NewdrA=="}' 2026-02-13T02:27:14.079 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b9e3d0f1-151c-4747-8a3c-057ae2476fe7 -i td/osd-scrub-snaps/0/new.json 2026-02-13T02:27:14.171 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:27:14.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-snaps/0/new.json 2026-02-13T02:27:14.180 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=b2b95964-f5a3-4b20-bd2b-d60f1ae5f8a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.453125/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQACjI5pKThNBBAArDAtEO8RYuAHnhq8NewdrA== --osd-uuid b9e3d0f1-151c-4747-8a3c-057ae2476fe7 2026-02-13T02:27:14.201 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:27:14.191+0000 7ff5acdaaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:27:14.202 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:27:14.191+0000 7ff5acdaaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:27:14.203 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:27:14.191+0000 7ff5acdaaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:27:14.204 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:27:14.191+0000 7ff5acdaaa40 -1 bdev(0x56310b9ccc00 td/osd-scrub-snaps/0/block) open stat got: (1) Operation not permitted 2026-02-13T02:27:14.204 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:27:14.191+0000 7ff5acdaaa40 -1 bluestore(td/osd-scrub-snaps/0) _read_fsid unparsable uuid 2026-02-13T02:27:14.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-snaps/0/keyring 2026-02-13T02:27:14.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:27:14.965 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T02:27:14.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-13T02:27:14.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-snaps/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:27:15.074 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T02:27:15.074 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-13T02:27:15.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=b2b95964-f5a3-4b20-bd2b-d60f1ae5f8a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.453125/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:27:15.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:27:15.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:27:15.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:27:15.107 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:27:15.095+0000 7f087b09aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:27:15.109 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:27:15.099+0000 7f087b09aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:27:15.110 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:27:15.099+0000 7f087b09aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:27:15.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-13T02:27:15.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:27:15.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:27:15.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:27:15.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:27:15.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:27:15.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:27:15.172 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:27:15.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:27:15.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:27:15.269 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:27:15.534 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:27:15.523+0000 7f087b09aa40 -1 Falling back to public interface 2026-02-13T02:27:16.270 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:27:16.271 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:27:16.271 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:27:16.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:27:16.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:27:16.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:27:16.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:27:16.526 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:27:16.515+0000 7f087b09aa40 -1 osd.0 0 log_to_monitors true 2026-02-13T02:27:17.369 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:27:17.369 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:27:17.369 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:27:17.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:27:17.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:27:17.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:27:17.537 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3189621497,v1:127.0.0.1:6801/3189621497] [v2:127.0.0.1:6802/3189621497,v1:127.0.0.1:6803/3189621497] exists,up b9e3d0f1-151c-4747-8a3c-057ae2476fe7 2026-02-13T02:27:17.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:27:17.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:27:17.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:27:17.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:27:17.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:744: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 1 2026-02-13T02:27:17.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-snaps 2026-02-13T02:27:17.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:27:17.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-13T02:27:17.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:27:17.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-snaps/1 2026-02-13T02:27:17.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=b2b95964-f5a3-4b20-bd2b-d60f1ae5f8a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2026-02-13T02:27:17.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:27:17.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:27:17.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:27:17.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2026-02-13T02:27:17.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2026-02-13T02:27:17.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:27:17.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:27:17.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2026-02-13T02:27:17.542 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:27:17.542 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:27:17.542 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:27:17.542 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:27:17.542 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:27:17.542 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.453125 2026-02-13T02:27:17.542 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.453125/$cluster-$name.asok' 2026-02-13T02:27:17.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.453125/$cluster-$name.asok' 2026-02-13T02:27:17.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:27:17.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:27:17.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:27:17.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2026-02-13T02:27:17.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2026-02-13T02:27:17.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:27:17.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:27:17.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:27:17.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:27:17.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:27:17.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:27:17.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-snaps/1 2026-02-13T02:27:17.544 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:27:17.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b10cf192-a605-401b-8aac-3f69afd06784 2026-02-13T02:27:17.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 b10cf192-a605-401b-8aac-3f69afd06784' 2026-02-13T02:27:17.545 INFO:tasks.workunit.client.0.trial112.stdout:add osd1 b10cf192-a605-401b-8aac-3f69afd06784 2026-02-13T02:27:17.545 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:27:17.557 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAFjI5pHr7ZIBAAbVAo3ikI4pG5MqK7UAMQpA== 2026-02-13T02:27:17.557 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAFjI5pHr7ZIBAAbVAo3ikI4pG5MqK7UAMQpA=="}' 2026-02-13T02:27:17.558 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b10cf192-a605-401b-8aac-3f69afd06784 -i td/osd-scrub-snaps/1/new.json 2026-02-13T02:27:17.727 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:27:17.737 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-snaps/1/new.json 2026-02-13T02:27:17.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=b2b95964-f5a3-4b20-bd2b-d60f1ae5f8a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.453125/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAFjI5pHr7ZIBAAbVAo3ikI4pG5MqK7UAMQpA== --osd-uuid b10cf192-a605-401b-8aac-3f69afd06784 2026-02-13T02:27:17.759 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:27:17.747+0000 7f640148da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:27:17.760 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:27:17.751+0000 7f640148da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:27:17.761 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:27:17.751+0000 7f640148da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:27:17.762 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:27:17.751+0000 7f640148da40 -1 bdev(0x55e030532c00 td/osd-scrub-snaps/1/block) open stat got: (1) Operation not permitted 2026-02-13T02:27:17.762 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:27:17.751+0000 7f640148da40 -1 bluestore(td/osd-scrub-snaps/1) _read_fsid unparsable uuid 2026-02-13T02:27:18.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-snaps/1/keyring 2026-02-13T02:27:18.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:27:18.416 INFO:tasks.workunit.client.0.trial112.stdout:adding osd1 key to auth repository 2026-02-13T02:27:18.416 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-13T02:27:18.416 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-snaps/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:27:18.621 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T02:27:18.621 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-13T02:27:18.621 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=b2b95964-f5a3-4b20-bd2b-d60f1ae5f8a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.453125/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:27:18.621 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:27:18.621 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:27:18.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:27:18.652 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:27:18.639+0000 7fb0f22a2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:27:18.653 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:27:18.643+0000 7fb0f22a2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:27:18.654 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:27:18.643+0000 7fb0f22a2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:27:18.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-13T02:27:18.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:27:18.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:27:18.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:27:18.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:27:18.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:27:18.782 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:27:18.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:27:18.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:27:18.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:27:18.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:27:19.045 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:27:19.035+0000 7fb0f22a2a40 -1 Falling back to public interface 2026-02-13T02:27:19.695 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:27:19.683+0000 7fb0f22a2a40 -1 osd.1 0 log_to_monitors true 2026-02-13T02:27:19.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:27:19.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:27:19.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:27:19.952 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:27:19.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:27:19.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:27:20.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:27:21.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:27:21.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:27:21.115 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:27:21.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:27:21.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:27:21.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:27:21.281 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2849951474,v1:127.0.0.1:6811/2849951474] [v2:127.0.0.1:6812/2849951474,v1:127.0.0.1:6813/2849951474] exists,up b10cf192-a605-401b-8aac-3f69afd06784 2026-02-13T02:27:21.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:27:21.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:27:21.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:27:21.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:748: _scrub_snaps_multi: ceph osd set noscrub 2026-02-13T02:27:21.490 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:27:21.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:749: _scrub_snaps_multi: ceph osd set nodeep-scrub 2026-02-13T02:27:21.695 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:27:21.705 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:752: _scrub_snaps_multi: create_pool test 1 1 2026-02-13T02:27:21.705 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2026-02-13T02:27:21.898 INFO:tasks.workunit.client.0.trial112.stderr:pool 'test' created 2026-02-13T02:27:21.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:27:22.910 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:753: _scrub_snaps_multi: wait_for_clean 2026-02-13T02:27:22.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:27:22.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:27:22.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:27:22.911 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:27:22.912 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:27:22.912 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:27:22.912 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:27:22.912 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:27:22.912 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:27:23.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:27:23.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:27:23.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:27:23.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:27:23.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:27:23.013 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:27:23.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:27:23.177 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:27:23.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:27:23.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:27:23.178 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:27:23.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-13T02:27:23.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-13T02:27:23.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-13T02:27:23.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:27:23.256 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:27:23.331 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705666 2026-02-13T02:27:23.331 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705666 2026-02-13T02:27:23.331 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705666' 2026-02-13T02:27:23.331 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:27:23.331 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-13T02:27:23.331 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:27:23.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:27:23.333 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-13T02:27:23.333 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:27:23.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-13T02:27:23.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-13T02:27:23.335 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836483 2026-02-13T02:27:23.335 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:27:23.507 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-13T02:27:23.507 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:27:24.509 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:27:24.509 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:27:24.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-13T02:27:24.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:27:24.675 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705666 2026-02-13T02:27:24.675 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:27:24.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:27:24.677 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705666 2026-02-13T02:27:24.677 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:27:24.678 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705666 2026-02-13T02:27:24.678 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705666 2026-02-13T02:27:24.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705666' 2026-02-13T02:27:24.679 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:27:24.852 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705666 -lt 38654705666 2026-02-13T02:27:24.853 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:27:24.853 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:27:24.853 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:27:25.054 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:27:25.054 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:27:25.054 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:27:25.054 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:27:25.054 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:27:25.055 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:27:25.055 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:27:25.055 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:27:25.222 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:27:25.223 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:27:25.223 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:27:25.223 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:27:25.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:27:25.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:27:25.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:27:25.427 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: ceph osd dump 2026-02-13T02:27:25.427 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: grep '^pool.*['\'']test['\'']' 2026-02-13T02:27:25.428 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: awk '{ print $2 }' 2026-02-13T02:27:25.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: poolid=1 2026-02-13T02:27:25.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:756: _scrub_snaps_multi: dd if=/dev/urandom of=testdata.453125 bs=1032 count=1 2026-02-13T02:27:25.594 INFO:tasks.workunit.client.0.trial112.stderr:1+0 records in 2026-02-13T02:27:25.594 INFO:tasks.workunit.client.0.trial112.stderr:1+0 records out 2026-02-13T02:27:25.594 INFO:tasks.workunit.client.0.trial112.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 5.6144e-05 s, 18.4 MB/s 2026-02-13T02:27:25.594 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: seq 1 16 2026-02-13T02:27:25.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-13T02:27:25.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj1 testdata.453125 2026-02-13T02:27:25.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-13T02:27:25.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj2 testdata.453125 2026-02-13T02:27:25.646 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-13T02:27:25.646 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj3 testdata.453125 2026-02-13T02:27:25.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-13T02:27:25.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj4 testdata.453125 2026-02-13T02:27:25.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-13T02:27:25.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj5 testdata.453125 2026-02-13T02:27:25.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-13T02:27:25.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj6 testdata.453125 2026-02-13T02:27:25.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-13T02:27:25.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj7 testdata.453125 2026-02-13T02:27:25.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-13T02:27:25.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj8 testdata.453125 2026-02-13T02:27:25.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-13T02:27:25.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj9 testdata.453125 2026-02-13T02:27:25.847 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-13T02:27:25.847 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj10 testdata.453125 2026-02-13T02:27:25.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-13T02:27:25.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj11 testdata.453125 2026-02-13T02:27:25.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-13T02:27:25.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj12 testdata.453125 2026-02-13T02:27:25.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-13T02:27:25.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj13 testdata.453125 2026-02-13T02:27:25.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-13T02:27:25.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj14 testdata.453125 2026-02-13T02:27:25.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-13T02:27:25.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj15 testdata.453125 2026-02-13T02:27:26.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-13T02:27:26.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj16 testdata.453125 2026-02-13T02:27:26.046 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:762: _scrub_snaps_multi: get_primary test obj1 2026-02-13T02:27:26.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2026-02-13T02:27:26.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-13T02:27:26.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2026-02-13T02:27:26.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T02:27:26.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:762: _scrub_snaps_multi: local primary=1 2026-02-13T02:27:26.216 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:763: _scrub_snaps_multi: get_not_primary test obj1 2026-02-13T02:27:26.216 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=test 2026-02-13T02:27:26.216 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=obj1 2026-02-13T02:27:26.217 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary test obj1 2026-02-13T02:27:26.217 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2026-02-13T02:27:26.217 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-13T02:27:26.217 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2026-02-13T02:27:26.217 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T02:27:26.383 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-13T02:27:26.384 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map test obj1 2026-02-13T02:27:26.384 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-13T02:27:26.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:763: _scrub_snaps_multi: local replica=0 2026-02-13T02:27:26.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:765: _scrub_snaps_multi: eval create_scenario td/osd-scrub-snaps test testdata.453125 '$primary' 2026-02-13T02:27:26.551 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:765: _scrub_snaps_multi: create_scenario td/osd-scrub-snaps test testdata.453125 1 2026-02-13T02:27:26.551 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:45: create_scenario: local dir=td/osd-scrub-snaps 2026-02-13T02:27:26.551 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:46: create_scenario: local poolname=test 2026-02-13T02:27:26.551 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:47: create_scenario: local TESTDATA=testdata.453125 2026-02-13T02:27:26.551 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:48: create_scenario: local osd=1 2026-02-13T02:27:26.551 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:50: create_scenario: SNAP=1 2026-02-13T02:27:26.551 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:51: create_scenario: rados -p test mksnap snap1 2026-02-13T02:27:26.623 INFO:tasks.workunit.client.0.trial112.stdout:created pool test snap snap1 2026-02-13T02:27:26.625 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:52: create_scenario: dd if=/dev/urandom of=testdata.453125 bs=256 count=1 2026-02-13T02:27:26.626 INFO:tasks.workunit.client.0.trial112.stderr:1+0 records in 2026-02-13T02:27:26.626 INFO:tasks.workunit.client.0.trial112.stderr:1+0 records out 2026-02-13T02:27:26.626 INFO:tasks.workunit.client.0.trial112.stderr:256 bytes copied, 8.6061e-05 s, 3.0 MB/s 2026-02-13T02:27:26.626 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:53: create_scenario: rados -p test put obj1 testdata.453125 2026-02-13T02:27:26.655 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:54: create_scenario: rados -p test put obj5 testdata.453125 2026-02-13T02:27:26.683 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:55: create_scenario: rados -p test put obj3 testdata.453125 2026-02-13T02:27:26.714 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: seq 6 14 2026-02-13T02:27:26.715 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-13T02:27:26.715 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj6 testdata.453125 2026-02-13T02:27:26.737 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-13T02:27:26.737 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj7 testdata.453125 2026-02-13T02:27:26.766 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-13T02:27:26.766 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj8 testdata.453125 2026-02-13T02:27:26.795 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-13T02:27:26.795 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj9 testdata.453125 2026-02-13T02:27:26.824 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-13T02:27:26.824 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj10 testdata.453125 2026-02-13T02:27:26.853 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-13T02:27:26.853 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj11 testdata.453125 2026-02-13T02:27:26.882 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-13T02:27:26.882 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj12 testdata.453125 2026-02-13T02:27:26.910 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-13T02:27:26.910 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj13 testdata.453125 2026-02-13T02:27:26.939 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-13T02:27:26.939 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj14 testdata.453125 2026-02-13T02:27:26.967 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:60: create_scenario: SNAP=2 2026-02-13T02:27:26.967 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:61: create_scenario: rados -p test mksnap snap2 2026-02-13T02:27:27.045 INFO:tasks.workunit.client.0.trial112.stdout:created pool test snap snap2 2026-02-13T02:27:27.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:62: create_scenario: dd if=/dev/urandom of=testdata.453125 bs=256 count=2 2026-02-13T02:27:27.048 INFO:tasks.workunit.client.0.trial112.stderr:2+0 records in 2026-02-13T02:27:27.048 INFO:tasks.workunit.client.0.trial112.stderr:2+0 records out 2026-02-13T02:27:27.048 INFO:tasks.workunit.client.0.trial112.stderr:512 bytes copied, 9.0819e-05 s, 5.6 MB/s 2026-02-13T02:27:27.048 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:63: create_scenario: rados -p test put obj5 testdata.453125 2026-02-13T02:27:27.078 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:65: create_scenario: SNAP=3 2026-02-13T02:27:27.078 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:66: create_scenario: rados -p test mksnap snap3 2026-02-13T02:27:27.145 INFO:tasks.workunit.client.0.trial112.stdout:created pool test snap snap3 2026-02-13T02:27:27.147 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:67: create_scenario: dd if=/dev/urandom of=testdata.453125 bs=256 count=3 2026-02-13T02:27:27.149 INFO:tasks.workunit.client.0.trial112.stderr:3+0 records in 2026-02-13T02:27:27.149 INFO:tasks.workunit.client.0.trial112.stderr:3+0 records out 2026-02-13T02:27:27.149 INFO:tasks.workunit.client.0.trial112.stderr:768 bytes copied, 9.2002e-05 s, 8.3 MB/s 2026-02-13T02:27:27.149 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:68: create_scenario: rados -p test put obj3 testdata.453125 2026-02-13T02:27:27.178 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:70: create_scenario: SNAP=4 2026-02-13T02:27:27.178 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:71: create_scenario: rados -p test mksnap snap4 2026-02-13T02:27:27.247 INFO:tasks.workunit.client.0.trial112.stdout:created pool test snap snap4 2026-02-13T02:27:27.250 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:72: create_scenario: dd if=/dev/urandom of=testdata.453125 bs=256 count=4 2026-02-13T02:27:27.251 INFO:tasks.workunit.client.0.trial112.stderr:4+0 records in 2026-02-13T02:27:27.251 INFO:tasks.workunit.client.0.trial112.stderr:4+0 records out 2026-02-13T02:27:27.251 INFO:tasks.workunit.client.0.trial112.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 9.636e-05 s, 10.6 MB/s 2026-02-13T02:27:27.251 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:73: create_scenario: rados -p test put obj5 testdata.453125 2026-02-13T02:27:27.282 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:74: create_scenario: rados -p test put obj2 testdata.453125 2026-02-13T02:27:27.310 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:76: create_scenario: SNAP=5 2026-02-13T02:27:27.310 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:77: create_scenario: rados -p test mksnap snap5 2026-02-13T02:27:27.350 INFO:tasks.workunit.client.0.trial112.stdout:created pool test snap snap5 2026-02-13T02:27:27.353 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:78: create_scenario: SNAP=6 2026-02-13T02:27:27.353 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:79: create_scenario: rados -p test mksnap snap6 2026-02-13T02:27:27.452 INFO:tasks.workunit.client.0.trial112.stdout:created pool test snap snap6 2026-02-13T02:27:27.454 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:80: create_scenario: dd if=/dev/urandom of=testdata.453125 bs=256 count=6 2026-02-13T02:27:27.455 INFO:tasks.workunit.client.0.trial112.stderr:6+0 records in 2026-02-13T02:27:27.455 INFO:tasks.workunit.client.0.trial112.stderr:6+0 records out 2026-02-13T02:27:27.455 INFO:tasks.workunit.client.0.trial112.stderr:1536 bytes (1.5 kB, 1.5 KiB) copied, 8.8355e-05 s, 17.4 MB/s 2026-02-13T02:27:27.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:81: create_scenario: rados -p test put obj5 testdata.453125 2026-02-13T02:27:27.486 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:83: create_scenario: SNAP=7 2026-02-13T02:27:27.486 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:84: create_scenario: rados -p test mksnap snap7 2026-02-13T02:27:27.554 INFO:tasks.workunit.client.0.trial112.stdout:created pool test snap snap7 2026-02-13T02:27:27.556 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:86: create_scenario: rados -p test rm obj4 2026-02-13T02:27:27.580 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:87: create_scenario: rados -p test rm obj16 2026-02-13T02:27:27.601 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:88: create_scenario: rados -p test rm obj2 2026-02-13T02:27:27.629 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:90: create_scenario: kill_daemons td/osd-scrub-snaps TERM osd 2026-02-13T02:27:27.629 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:27:27.629 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:27:27.629 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:27:27.629 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:27:27.629 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:27:27.938 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:27:27.938 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj1 2026-02-13T02:27:28.108 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: JSON='["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:27:28.108 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:95: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' --force remove 2026-02-13T02:27:28.340 INFO:tasks.workunit.client.0.trial112.stdout:WARNING: only removing head with clones present 2026-02-13T02:27:28.340 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:ff7b1f36:::obj1:head# 2026-02-13T02:27:28.441 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj5 2026-02-13T02:27:28.441 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: grep '"snapid":2' 2026-02-13T02:27:28.597 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:27:28.597 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:98: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2026-02-13T02:27:28.793 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:c52c9666:::obj5:2# 2026-02-13T02:27:28.889 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj5 2026-02-13T02:27:28.889 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: grep '"snapid":1' 2026-02-13T02:27:29.052 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:27:29.052 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:101: create_scenario: OBJ5SAVE='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:27:29.052 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:103: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/1 list 2026-02-13T02:27:29.280 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:104: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2026-02-13T02:27:29.281 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:27:29.281 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2026-02-13T02:27:29.281 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:27:29.281 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-13T02:27:29.281 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2026-02-13T02:27:29.281 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2026-02-13T02:27:29.282 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2026-02-13T02:27:29.282 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2026-02-13T02:27:29.282 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2026-02-13T02:27:29.282 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2026-02-13T02:27:29.282 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:27:29.282 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2026-02-13T02:27:29.282 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2026-02-13T02:27:29.282 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2026-02-13T02:27:29.282 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2026-02-13T02:27:29.282 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:27:29.282 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2026-02-13T02:27:29.282 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:27:29.283 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:27:29.283 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:27:29.283 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2026-02-13T02:27:29.283 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:27:29.283 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2026-02-13T02:27:29.283 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:27:29.283 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:27:29.283 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:27:29.283 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2026-02-13T02:27:29.283 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:27:29.283 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:27:29.283 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:27:29.284 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2026-02-13T02:27:29.284 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2026-02-13T02:27:29.284 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:27:29.284 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:27:29.284 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2026-02-13T02:27:29.284 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2026-02-13T02:27:29.284 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:27:29.284 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:27:29.284 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2026-02-13T02:27:29.285 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:105: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2026-02-13T02:27:29.285 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-13T02:27:29.285 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:106: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --rmtype nosnapmap '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2026-02-13T02:27:29.487 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:c52c9666:::obj5:1# 2026-02-13T02:27:29.584 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:108: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/1 list 2026-02-13T02:27:29.799 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:109: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2026-02-13T02:27:29.801 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:27:29.801 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2026-02-13T02:27:29.801 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:27:29.801 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-13T02:27:29.801 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2026-02-13T02:27:29.801 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2026-02-13T02:27:29.801 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2026-02-13T02:27:29.801 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2026-02-13T02:27:29.801 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2026-02-13T02:27:29.801 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2026-02-13T02:27:29.801 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:27:29.802 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2026-02-13T02:27:29.802 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2026-02-13T02:27:29.802 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2026-02-13T02:27:29.802 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2026-02-13T02:27:29.802 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:27:29.802 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2026-02-13T02:27:29.802 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:27:29.802 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:27:29.802 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:27:29.802 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2026-02-13T02:27:29.802 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:27:29.802 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2026-02-13T02:27:29.803 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:27:29.803 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:27:29.803 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:27:29.803 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2026-02-13T02:27:29.803 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:27:29.803 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:27:29.803 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:27:29.803 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2026-02-13T02:27:29.803 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2026-02-13T02:27:29.803 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:27:29.803 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:27:29.803 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2026-02-13T02:27:29.803 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2026-02-13T02:27:29.804 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:27:29.804 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:27:29.804 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2026-02-13T02:27:29.804 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:110: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2026-02-13T02:27:29.804 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-13T02:27:29.805 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:111: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2026-02-13T02:27:29.805 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj5 2026-02-13T02:27:29.805 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: grep '"snapid":4' 2026-02-13T02:27:29.944 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:27:29.944 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:114: create_scenario: dd if=/dev/urandom of=testdata.453125 bs=256 count=18 2026-02-13T02:27:29.945 INFO:tasks.workunit.client.0.trial112.stderr:18+0 records in 2026-02-13T02:27:29.945 INFO:tasks.workunit.client.0.trial112.stderr:18+0 records out 2026-02-13T02:27:29.945 INFO:tasks.workunit.client.0.trial112.stderr:4608 bytes (4.6 kB, 4.5 KiB) copied, 9.2402e-05 s, 49.9 MB/s 2026-02-13T02:27:29.945 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:115: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.453125 2026-02-13T02:27:30.236 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj3 2026-02-13T02:27:30.427 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: JSON='["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:27:30.427 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:118: create_scenario: dd if=/dev/urandom of=testdata.453125 bs=256 count=15 2026-02-13T02:27:30.428 INFO:tasks.workunit.client.0.trial112.stderr:15+0 records in 2026-02-13T02:27:30.429 INFO:tasks.workunit.client.0.trial112.stderr:15+0 records out 2026-02-13T02:27:30.429 INFO:tasks.workunit.client.0.trial112.stderr:3840 bytes (3.8 kB, 3.8 KiB) copied, 0.000321399 s, 11.9 MB/s 2026-02-13T02:27:30.429 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:119: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.453125 2026-02-13T02:27:30.681 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj4 2026-02-13T02:27:30.681 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: grep '"snapid":7' 2026-02-13T02:27:30.856 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: JSON='["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:27:30.856 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:122: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' remove 2026-02-13T02:27:31.023 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:0ee9ae15:::obj4:7# 2026-02-13T02:27:31.116 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:125: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/1 list 2026-02-13T02:27:31.343 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:126: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2026-02-13T02:27:31.344 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2026-02-13T02:27:31.345 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:27:31.345 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-13T02:27:31.345 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2026-02-13T02:27:31.345 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2026-02-13T02:27:31.345 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2026-02-13T02:27:31.345 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2026-02-13T02:27:31.345 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2026-02-13T02:27:31.345 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2026-02-13T02:27:31.345 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:27:31.345 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2026-02-13T02:27:31.345 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2026-02-13T02:27:31.345 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2026-02-13T02:27:31.345 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2026-02-13T02:27:31.345 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2026-02-13T02:27:31.345 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:27:31.345 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:27:31.345 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2026-02-13T02:27:31.345 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:27:31.345 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2026-02-13T02:27:31.345 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:27:31.345 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:27:31.345 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2026-02-13T02:27:31.345 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:27:31.345 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:27:31.346 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2026-02-13T02:27:31.346 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2026-02-13T02:27:31.346 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:27:31.346 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2026-02-13T02:27:31.346 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2026-02-13T02:27:31.346 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:27:31.346 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2026-02-13T02:27:31.346 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:127: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2026-02-13T02:27:31.346 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:27:31.346 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:27:31.346 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:27:31.346 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:27:31.346 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:27:31.346 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:27:31.346 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:27:31.346 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj16 2026-02-13T02:27:31.347 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: grep '"snapid":7' 2026-02-13T02:27:31.512 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: JSON='["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:27:31.512 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:129: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --rmtype snapmap '["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' remove 2026-02-13T02:27:31.692 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:461f8b5e:::obj16:7# 2026-02-13T02:27:31.781 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:131: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/1 list 2026-02-13T02:27:32.004 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:132: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2026-02-13T02:27:32.005 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2026-02-13T02:27:32.005 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-13T02:27:32.005 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2026-02-13T02:27:32.005 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2026-02-13T02:27:32.005 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2026-02-13T02:27:32.005 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2026-02-13T02:27:32.005 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2026-02-13T02:27:32.005 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2026-02-13T02:27:32.005 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:27:32.005 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2026-02-13T02:27:32.005 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2026-02-13T02:27:32.006 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2026-02-13T02:27:32.006 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2026-02-13T02:27:32.006 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2026-02-13T02:27:32.006 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:27:32.006 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2026-02-13T02:27:32.006 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2026-02-13T02:27:32.006 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:27:32.006 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2026-02-13T02:27:32.006 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:27:32.006 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2026-02-13T02:27:32.006 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2026-02-13T02:27:32.006 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2026-02-13T02:27:32.006 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2026-02-13T02:27:32.006 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2026-02-13T02:27:32.006 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:133: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2026-02-13T02:27:32.006 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:134: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2026-02-13T02:27:32.008 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj2 2026-02-13T02:27:32.191 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: JSON='["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:27:32.191 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:137: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' rm-attr snapset 2026-02-13T02:27:32.457 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: echo '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:27:32.457 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: sed 's/snapid":1/snapid":7/' 2026-02-13T02:27:32.459 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:27:32.459 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:141: create_scenario: dd if=/dev/urandom of=testdata.453125 bs=256 count=7 2026-02-13T02:27:32.460 INFO:tasks.workunit.client.0.trial112.stderr:7+0 records in 2026-02-13T02:27:32.460 INFO:tasks.workunit.client.0.trial112.stderr:7+0 records out 2026-02-13T02:27:32.460 INFO:tasks.workunit.client.0.trial112.stderr:1792 bytes (1.8 kB, 1.8 KiB) copied, 6.6674e-05 s, 26.9 MB/s 2026-02-13T02:27:32.460 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:142: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.453125 2026-02-13T02:27:32.762 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj6 2026-02-13T02:27:32.843 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-13T02:27:32.935 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: JSON='["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:27:32.935 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:145: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset 2026-02-13T02:27:33.205 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj7 2026-02-13T02:27:33.281 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-13T02:27:33.371 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: JSON='["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:27:33.371 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:147: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset corrupt 2026-02-13T02:27:33.653 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj8 2026-02-13T02:27:33.727 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-13T02:27:33.839 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: JSON='["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:27:33.839 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:149: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset seq 2026-02-13T02:27:34.136 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj9 2026-02-13T02:27:34.203 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-13T02:27:34.295 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: JSON='["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:27:34.295 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:151: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_size 2026-02-13T02:27:34.613 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj10 2026-02-13T02:27:34.698 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-13T02:27:34.799 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: JSON='["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:27:34.799 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:153: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_overlap 2026-02-13T02:27:35.080 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj11 2026-02-13T02:27:35.138 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-13T02:27:35.247 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: JSON='["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:27:35.247 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:155: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clones 2026-02-13T02:27:35.520 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj12 2026-02-13T02:27:35.599 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-13T02:27:35.676 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: JSON='["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:27:35.676 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:157: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset head 2026-02-13T02:27:35.941 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj13 2026-02-13T02:27:36.015 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-13T02:27:36.116 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: JSON='["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:27:36.116 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:159: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset snaps 2026-02-13T02:27:36.393 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj14 2026-02-13T02:27:36.476 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-13T02:27:36.576 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: JSON='["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:27:36.576 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:161: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset size 2026-02-13T02:27:36.848 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:163: create_scenario: echo garbage 2026-02-13T02:27:36.849 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj15 2026-02-13T02:27:36.931 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-13T02:27:37.019 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: JSON='["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:27:37.020 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:165: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' set-attr snapset td/osd-scrub-snaps/bad 2026-02-13T02:27:37.296 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:166: create_scenario: rm -f td/osd-scrub-snaps/bad 2026-02-13T02:27:37.297 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:167: create_scenario: return 0 2026-02-13T02:27:37.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:767: _scrub_snaps_multi: rm -f testdata.453125 2026-02-13T02:27:37.299 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: expr 2 - 1 2026-02-13T02:27:37.300 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: seq 0 1 2026-02-13T02:27:37.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:27:37.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:771: _scrub_snaps_multi: activate_osd td/osd-scrub-snaps 0 2026-02-13T02:27:37.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-snaps 2026-02-13T02:27:37.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:27:37.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:27:37.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:27:37.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-snaps/0 2026-02-13T02:27:37.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=b2b95964-f5a3-4b20-bd2b-d60f1ae5f8a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2026-02-13T02:27:37.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:27:37.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:27:37.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:27:37.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2026-02-13T02:27:37.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2026-02-13T02:27:37.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:27:37.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:27:37.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2026-02-13T02:27:37.304 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:27:37.304 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:27:37.304 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:27:37.304 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:27:37.305 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:27:37.305 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.453125 2026-02-13T02:27:37.305 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.453125/$cluster-$name.asok' 2026-02-13T02:27:37.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.453125/$cluster-$name.asok' 2026-02-13T02:27:37.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:27:37.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2026-02-13T02:27:37.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2026-02-13T02:27:37.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:27:37.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:27:37.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:27:37.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:27:37.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:27:37.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:27:37.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-snaps/0 2026-02-13T02:27:37.307 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T02:27:37.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:27:37.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=b2b95964-f5a3-4b20-bd2b-d60f1ae5f8a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.453125/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:27:37.308 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-snaps/0/whoami 2026-02-13T02:27:37.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:27:37.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:27:37.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:27:37.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:27:37.330 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:27:37.322+0000 7f1550fd6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:27:37.331 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:27:37.322+0000 7f1550fd6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:27:37.332 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:27:37.326+0000 7f1550fd6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:27:37.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:27:37.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:27:37.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:27:37.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:27:37.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:27:37.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:27:37.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:27:37.471 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:27:37.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:27:37.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:27:37.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:27:37.734 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:27:37.726+0000 7f1550fd6a40 -1 Falling back to public interface 2026-02-13T02:27:38.384 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:27:38.378+0000 7f1550fd6a40 -1 osd.0 22 log_to_monitors true 2026-02-13T02:27:38.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:27:38.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:27:38.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:27:38.639 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:27:38.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:27:38.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:27:38.806 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 25 up_thru 25 down_at 23 last_clean_interval [5,22) [v2:127.0.0.1:6800/2993843151,v1:127.0.0.1:6801/2993843151] [v2:127.0.0.1:6802/2993843151,v1:127.0.0.1:6803/2993843151] exists,up b9e3d0f1-151c-4747-8a3c-057ae2476fe7 2026-02-13T02:27:38.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:27:38.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:27:38.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:27:38.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:27:38.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:771: _scrub_snaps_multi: activate_osd td/osd-scrub-snaps 1 2026-02-13T02:27:38.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-snaps 2026-02-13T02:27:38.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:27:38.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:27:38.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:27:38.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-snaps/1 2026-02-13T02:27:38.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=b2b95964-f5a3-4b20-bd2b-d60f1ae5f8a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2026-02-13T02:27:38.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:27:38.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:27:38.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:27:38.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2026-02-13T02:27:38.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2026-02-13T02:27:38.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:27:38.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:27:38.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2026-02-13T02:27:38.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:27:38.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:27:38.812 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:27:38.812 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:27:38.812 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:27:38.812 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.453125 2026-02-13T02:27:38.812 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.453125/$cluster-$name.asok' 2026-02-13T02:27:38.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.453125/$cluster-$name.asok' 2026-02-13T02:27:38.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:27:38.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2026-02-13T02:27:38.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2026-02-13T02:27:38.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:27:38.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:27:38.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:27:38.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:27:38.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:27:38.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:27:38.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-snaps/1 2026-02-13T02:27:38.813 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T02:27:38.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:27:38.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=b2b95964-f5a3-4b20-bd2b-d60f1ae5f8a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.453125/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:27:38.814 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-snaps/1/whoami 2026-02-13T02:27:38.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:27:38.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:27:38.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:27:38.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:27:38.833 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:27:38.826+0000 7fd1c4309a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:27:38.834 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:27:38.826+0000 7fd1c4309a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:27:38.836 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:27:38.826+0000 7fd1c4309a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:27:38.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:27:38.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:27:38.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:27:38.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:27:38.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:27:38.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:27:38.974 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:27:38.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:27:38.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:27:38.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:27:39.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:27:39.178 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:27:39.170+0000 7fd1c4309a40 -1 Falling back to public interface 2026-02-13T02:27:39.789 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:27:39.782+0000 7fd1c4309a40 -1 osd.1 22 log_to_monitors true 2026-02-13T02:27:40.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:27:40.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:27:40.148 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:27:40.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:27:40.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:27:40.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:27:40.310 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 29 up_thru 29 down_at 23 last_clean_interval [9,22) [v2:127.0.0.1:6810/2935222265,v1:127.0.0.1:6811/2935222265] [v2:127.0.0.1:6812/2935222265,v1:127.0.0.1:6813/2935222265] exists,up b10cf192-a605-401b-8aac-3f69afd06784 2026-02-13T02:27:40.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:27:40.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:27:40.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:27:40.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:774: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_max 3 2026-02-13T02:27:40.381 INFO:tasks.workunit.client.0.trial112.stdout:osd.0: { 2026-02-13T02:27:40.381 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-13T02:27:40.381 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:27:40.389 INFO:tasks.workunit.client.0.trial112.stdout:osd.1: { 2026-02-13T02:27:40.389 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-13T02:27:40.389 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:27:40.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:775: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_min 3 2026-02-13T02:27:40.477 INFO:tasks.workunit.client.0.trial112.stdout:osd.0: { 2026-02-13T02:27:40.477 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_shallow_scrub_chunk_min = '' " 2026-02-13T02:27:40.477 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:27:40.484 INFO:tasks.workunit.client.0.trial112.stdout:osd.1: { 2026-02-13T02:27:40.484 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_shallow_scrub_chunk_min = '' " 2026-02-13T02:27:40.484 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:27:40.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:776: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_scrub_chunk_min 3 2026-02-13T02:27:40.572 INFO:tasks.workunit.client.0.trial112.stdout:osd.0: { 2026-02-13T02:27:40.572 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_chunk_min = '' " 2026-02-13T02:27:40.572 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:27:40.580 INFO:tasks.workunit.client.0.trial112.stdout:osd.1: { 2026-02-13T02:27:40.580 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_chunk_min = '' " 2026-02-13T02:27:40.580 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:27:40.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:777: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_seconds 1 2026-02-13T02:27:40.665 INFO:tasks.workunit.client.0.trial112.stdout:osd.0: { 2026-02-13T02:27:40.666 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_pg_stat_report_interval_max_seconds = '' " 2026-02-13T02:27:40.666 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:27:40.673 INFO:tasks.workunit.client.0.trial112.stdout:osd.1: { 2026-02-13T02:27:40.673 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_pg_stat_report_interval_max_seconds = '' " 2026-02-13T02:27:40.673 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:27:40.680 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:778: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_epochs 1 2026-02-13T02:27:40.758 INFO:tasks.workunit.client.0.trial112.stdout:osd.0: { 2026-02-13T02:27:40.758 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_pg_stat_report_interval_max_epochs = '' " 2026-02-13T02:27:40.758 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:27:40.765 INFO:tasks.workunit.client.0.trial112.stdout:osd.1: { 2026-02-13T02:27:40.766 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_pg_stat_report_interval_max_epochs = '' " 2026-02-13T02:27:40.766 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:27:40.773 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:779: _scrub_snaps_multi: wait_for_clean 2026-02-13T02:27:40.774 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:27:40.774 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:27:40.774 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:27:40.774 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:27:40.775 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:27:40.775 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:27:40.775 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:27:40.775 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:27:40.775 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:27:40.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:27:40.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:27:40.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:27:40.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:27:40.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:27:40.890 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:27:41.052 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:27:41.052 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:27:41.053 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:27:41.053 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:27:41.053 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:27:41.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=107374182402 2026-02-13T02:27:41.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 107374182402 2026-02-13T02:27:41.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-107374182402' 2026-02-13T02:27:41.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:27:41.133 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:27:41.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=124554051586 2026-02-13T02:27:41.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 124554051586 2026-02-13T02:27:41.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-107374182402 1-124554051586' 2026-02-13T02:27:41.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:27:41.213 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-107374182402 2026-02-13T02:27:41.213 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:27:41.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:27:41.215 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-107374182402 2026-02-13T02:27:41.215 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:27:41.217 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=107374182402 2026-02-13T02:27:41.217 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2026-02-13T02:27:41.217 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 107374182402 2026-02-13T02:27:41.217 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:27:41.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 107374182402 2026-02-13T02:27:41.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:27:42.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:27:42.379 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:27:42.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 107374182402 -lt 107374182402 2026-02-13T02:27:42.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:27:42.550 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-124554051586 2026-02-13T02:27:42.550 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:27:42.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:27:42.552 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-124554051586 2026-02-13T02:27:42.552 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:27:42.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=124554051586 2026-02-13T02:27:42.554 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 124554051586 2026-02-13T02:27:42.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 124554051586' 2026-02-13T02:27:42.554 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:27:42.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 124554051586 -lt 124554051586 2026-02-13T02:27:42.728 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:27:42.729 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:27:42.729 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:27:42.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:27:42.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:27:42.936 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:27:42.936 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:27:42.936 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:27:42.936 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:27:42.936 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:27:42.937 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:27:43.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:27:43.103 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:27:43.103 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:27:43.103 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:27:43.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:27:43.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:27:43.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:27:43.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:781: _scrub_snaps_multi: local pgid=1.0 2026-02-13T02:27:43.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:782: _scrub_snaps_multi: pg_scrub 1.0 2026-02-13T02:27:43.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=1.0 2026-02-13T02:27:43.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 1.0 2026-02-13T02:27:43.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=1.0 2026-02-13T02:27:43.303 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-13T02:27:43.303 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:27:43.304 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:27:43.304 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:27:43.304 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:27:43.304 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:27:43.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-13T02:27:43.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-13T02:27:43.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-13T02:27:43.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-13T02:27:43.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:27:43.560 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:27:43.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:27:43.727 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:27:43.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:27:43.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:27:43.728 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:27:43.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=107374182404 2026-02-13T02:27:43.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 107374182404 2026-02-13T02:27:43.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-107374182404' 2026-02-13T02:27:43.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:27:43.807 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:27:43.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=124554051587 2026-02-13T02:27:43.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 124554051587 2026-02-13T02:27:43.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-107374182404 1-124554051587' 2026-02-13T02:27:43.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:27:43.884 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-107374182404 2026-02-13T02:27:43.884 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:27:43.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:27:43.886 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-107374182404 2026-02-13T02:27:43.886 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:27:43.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=107374182404 2026-02-13T02:27:43.887 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 107374182404 2026-02-13T02:27:43.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 107374182404' 2026-02-13T02:27:43.888 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:27:44.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 107374182402 -lt 107374182404 2026-02-13T02:27:44.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:27:45.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:27:45.070 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:27:45.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 107374182404 -lt 107374182404 2026-02-13T02:27:45.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:27:45.236 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-124554051587 2026-02-13T02:27:45.236 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:27:45.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:27:45.238 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-124554051587 2026-02-13T02:27:45.239 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:27:45.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=124554051587 2026-02-13T02:27:45.240 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 124554051587 2026-02-13T02:27:45.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 124554051587' 2026-02-13T02:27:45.241 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:27:45.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 124554051587 -lt 124554051587 2026-02-13T02:27:45.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-13T02:27:45.406 INFO:tasks.workunit.client.0.trial112.stdout:#---------- 1.0 loop 0 2026-02-13T02:27:45.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 0' 2026-02-13T02:27:45.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-13T02:27:45.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-13T02:27:45.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-13T02:27:45.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-13T02:27:45.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-13T02:27:45.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-13T02:27:45.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-13T02:27:45.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-13T02:27:45.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-13T02:27:45.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: get_last_scrub_stamp 1.0 2026-02-13T02:27:45.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:27:45.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:27:45.489 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:27:45.489 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:27:45.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: local last_scrub=2026-02-13T02:27:21.891155+0000 2026-02-13T02:27:45.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_scrub: ceph pg scrub 1.0 2026-02-13T02:27:45.810 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 1.0 on osd.1 to scrub 2026-02-13T02:27:45.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_scrub: wait_for_scrub 1.0 2026-02-13T02:27:21.891155+0000 2026-02-13T02:27:45.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-13T02:27:45.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:27:21.891155+0000 2026-02-13T02:27:45.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:27:45.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:27:45.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:27:45.822 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:27:45.822 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:27:45.822 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:27:45.822 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:27:45.822 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:27:45.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:27:21.891155+0000 '>' 2026-02-13T02:27:21.891155+0000 2026-02-13T02:27:45.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:27:46.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:27:46.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:27:46.984 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:27:46.985 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:27:46.985 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:27:46.985 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:27:46.985 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:27:47.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:27:21.891155+0000 '>' 2026-02-13T02:27:21.891155+0000 2026-02-13T02:27:47.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:27:48.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:27:48.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:27:48.149 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:27:48.149 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:27:48.149 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:27:48.149 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:27:48.149 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:27:48.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:27:21.891155+0000 '>' 2026-02-13T02:27:21.891155+0000 2026-02-13T02:27:48.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:27:49.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:27:49.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:27:49.313 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:27:49.313 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:27:49.314 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:27:49.314 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:27:49.314 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:27:49.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:27:21.891155+0000 '>' 2026-02-13T02:27:21.891155+0000 2026-02-13T02:27:49.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:27:50.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:27:50.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:27:50.477 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:27:50.477 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:27:50.477 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:27:50.477 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:27:50.477 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:27:50.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:27:46.733768+0000 '>' 2026-02-13T02:27:21.891155+0000 2026-02-13T02:27:50.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:27:50.638 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: grep '_scan_snaps start' td/osd-scrub-snaps/osd.1.log 2026-02-13T02:27:50.638 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: wc -l 2026-02-13T02:27:50.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: test 14 -gt 3 2026-02-13T02:27:50.642 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: grep '_scan_snaps start' td/osd-scrub-snaps/osd.0.log 2026-02-13T02:27:50.642 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: wc -l 2026-02-13T02:27:50.645 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: test 14 -gt 3 2026-02-13T02:27:50.645 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:789: _scrub_snaps_multi: rados list-inconsistent-pg test 2026-02-13T02:27:50.677 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:791: _scrub_snaps_multi: jq '. | length' td/osd-scrub-snaps/json 2026-02-13T02:27:50.692 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:791: _scrub_snaps_multi: test 1 = 1 2026-02-13T02:27:50.693 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:793: _scrub_snaps_multi: jq -r '.[0]' td/osd-scrub-snaps/json 2026-02-13T02:27:50.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:793: _scrub_snaps_multi: test 1.0 = 1.0 2026-02-13T02:27:50.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:795: _scrub_snaps_multi: rados list-inconsistent-obj 1.0 --format=json-pretty 2026-02-13T02:27:50.722 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:27:50.722 INFO:tasks.workunit.client.0.trial112.stdout: "epoch": 29, 2026-02-13T02:27:50.722 INFO:tasks.workunit.client.0.trial112.stdout: "inconsistents": [ 2026-02-13T02:27:50.723 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.723 INFO:tasks.workunit.client.0.trial112.stdout: "object": { 2026-02-13T02:27:50.723 INFO:tasks.workunit.client.0.trial112.stdout: "name": "obj1", 2026-02-13T02:27:50.723 INFO:tasks.workunit.client.0.trial112.stdout: "nspace": "", 2026-02-13T02:27:50.723 INFO:tasks.workunit.client.0.trial112.stdout: "locator": "", 2026-02-13T02:27:50.723 INFO:tasks.workunit.client.0.trial112.stdout: "snap": "head", 2026-02-13T02:27:50.723 INFO:tasks.workunit.client.0.trial112.stdout: "version": 18 2026-02-13T02:27:50.723 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.723 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:27:50.723 INFO:tasks.workunit.client.0.trial112.stdout: "union_shard_errors": [ 2026-02-13T02:27:50.723 INFO:tasks.workunit.client.0.trial112.stdout: "missing" 2026-02-13T02:27:50.723 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:27:50.723 INFO:tasks.workunit.client.0.trial112.stdout: "selected_object_info": { 2026-02-13T02:27:50.723 INFO:tasks.workunit.client.0.trial112.stdout: "oid": { 2026-02-13T02:27:50.723 INFO:tasks.workunit.client.0.trial112.stdout: "oid": "obj1", 2026-02-13T02:27:50.723 INFO:tasks.workunit.client.0.trial112.stdout: "key": "", 2026-02-13T02:27:50.723 INFO:tasks.workunit.client.0.trial112.stdout: "snapid": -2, 2026-02-13T02:27:50.723 INFO:tasks.workunit.client.0.trial112.stdout: "hash": 1828249343, 2026-02-13T02:27:50.723 INFO:tasks.workunit.client.0.trial112.stdout: "max": 0, 2026-02-13T02:27:50.723 INFO:tasks.workunit.client.0.trial112.stdout: "pool": 1, 2026-02-13T02:27:50.723 INFO:tasks.workunit.client.0.trial112.stdout: "namespace": "" 2026-02-13T02:27:50.723 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.723 INFO:tasks.workunit.client.0.trial112.stdout: "version": "16'18", 2026-02-13T02:27:50.723 INFO:tasks.workunit.client.0.trial112.stdout: "prior_version": "15'1", 2026-02-13T02:27:50.723 INFO:tasks.workunit.client.0.trial112.stdout: "last_reqid": "client.4222.0:1", 2026-02-13T02:27:50.723 INFO:tasks.workunit.client.0.trial112.stdout: "user_version": 18, 2026-02-13T02:27:50.723 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:27:50.723 INFO:tasks.workunit.client.0.trial112.stdout: "mtime": "2026-02-13T02:27:26.646745+0000", 2026-02-13T02:27:50.723 INFO:tasks.workunit.client.0.trial112.stdout: "local_mtime": "2026-02-13T02:27:26.647217+0000", 2026-02-13T02:27:50.723 INFO:tasks.workunit.client.0.trial112.stdout: "lost": 0, 2026-02-13T02:27:50.723 INFO:tasks.workunit.client.0.trial112.stdout: "flags": [ 2026-02-13T02:27:50.723 INFO:tasks.workunit.client.0.trial112.stdout: "dirty", 2026-02-13T02:27:50.723 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest" 2026-02-13T02:27:50.723 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:27:50.723 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_seq": 0, 2026-02-13T02:27:50.723 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_size": 0, 2026-02-13T02:27:50.723 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest": "0xc6ceb5ce", 2026-02-13T02:27:50.723 INFO:tasks.workunit.client.0.trial112.stdout: "omap_digest": "0xffffffff", 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: "expected_object_size": 0, 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: "expected_write_size": 0, 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: "alloc_hint_flags": 0, 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: "manifest": { 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: "type": 0 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: "watchers": {}, 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: "shard_versions": [] 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: "shards": [ 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 0, 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: "primary": false, 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 1, 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: "primary": true, 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [ 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: "missing" 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: "object": { 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: "name": "obj10", 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: "nspace": "", 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: "locator": "", 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: "snap": "head", 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: "version": 32 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [ 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: "snapset_inconsistency" 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: "union_shard_errors": [], 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: "selected_object_info": { 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: "oid": { 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: "oid": "obj10", 2026-02-13T02:27:50.724 INFO:tasks.workunit.client.0.trial112.stdout: "key": "", 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: "snapid": -2, 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: "hash": 718195851, 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: "max": 0, 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: "pool": 1, 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: "namespace": "" 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: "version": "16'32", 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: "prior_version": "15'10", 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: "last_reqid": "client.4243.0:1", 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: "user_version": 32, 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: "mtime": "2026-02-13T02:27:26.845002+0000", 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: "local_mtime": "2026-02-13T02:27:26.845378+0000", 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: "lost": 0, 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: "flags": [ 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: "dirty", 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest" 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_seq": 0, 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_size": 0, 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest": "0xc6ceb5ce", 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: "omap_digest": "0xffffffff", 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: "expected_object_size": 0, 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: "expected_write_size": 0, 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: "alloc_hint_flags": 0, 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: "manifest": { 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: "type": 0 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: "watchers": {}, 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: "shard_versions": [] 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: "shards": [ 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 0, 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: "primary": false, 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: "snapset": { 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 1, 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: "clones": [ 2026-02-13T02:27:50.725 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: "snap": 1, 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: "size": 1032, 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: "overlap": "[]", 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: "snaps": [ 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: 1 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 1, 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: "primary": true, 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: "snapset": { 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 1, 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: "clones": [ 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: "snap": 1, 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: "size": 1032, 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: "overlap": "????", 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: "snaps": [ 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: 1 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: "object": { 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: "name": "obj11", 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: "nspace": "", 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: "locator": "", 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: "snap": "head", 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: "version": 34 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [ 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: "snapset_inconsistency" 2026-02-13T02:27:50.726 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: "union_shard_errors": [], 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: "selected_object_info": { 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: "oid": { 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: "oid": "obj11", 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: "key": "", 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: "snapid": -2, 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: "hash": 693400951, 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: "max": 0, 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: "pool": 1, 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: "namespace": "" 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: "version": "16'34", 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: "prior_version": "15'11", 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: "last_reqid": "client.4246.0:1", 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: "user_version": 34, 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: "mtime": "2026-02-13T02:27:26.874019+0000", 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: "local_mtime": "2026-02-13T02:27:26.874367+0000", 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: "lost": 0, 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: "flags": [ 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: "dirty", 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest" 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_seq": 0, 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_size": 0, 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest": "0xc6ceb5ce", 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: "omap_digest": "0xffffffff", 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: "expected_object_size": 0, 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: "expected_write_size": 0, 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: "alloc_hint_flags": 0, 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: "manifest": { 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: "type": 0 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: "watchers": {}, 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: "shard_versions": [] 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: "shards": [ 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 0, 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: "primary": false, 2026-02-13T02:27:50.727 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: "snapset": { 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 1, 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: "clones": [ 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: "snap": 1, 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: "size": 1032, 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: "overlap": "[]", 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: "snaps": [ 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: 1 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 1, 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: "primary": true, 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: "snapset": { 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 1, 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: "clones": [] 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: "object": { 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: "name": "obj13", 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: "nspace": "", 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: "locator": "", 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: "snap": "head", 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: "version": 38 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [ 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: "snapset_inconsistency" 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: "union_shard_errors": [], 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: "selected_object_info": { 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: "oid": { 2026-02-13T02:27:50.728 INFO:tasks.workunit.client.0.trial112.stdout: "oid": "obj13", 2026-02-13T02:27:50.729 INFO:tasks.workunit.client.0.trial112.stdout: "key": "", 2026-02-13T02:27:50.729 INFO:tasks.workunit.client.0.trial112.stdout: "snapid": -2, 2026-02-13T02:27:50.729 INFO:tasks.workunit.client.0.trial112.stdout: "hash": 2087409765, 2026-02-13T02:27:50.729 INFO:tasks.workunit.client.0.trial112.stdout: "max": 0, 2026-02-13T02:27:50.729 INFO:tasks.workunit.client.0.trial112.stdout: "pool": 1, 2026-02-13T02:27:50.729 INFO:tasks.workunit.client.0.trial112.stdout: "namespace": "" 2026-02-13T02:27:50.729 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.729 INFO:tasks.workunit.client.0.trial112.stdout: "version": "16'38", 2026-02-13T02:27:50.729 INFO:tasks.workunit.client.0.trial112.stdout: "prior_version": "15'13", 2026-02-13T02:27:50.729 INFO:tasks.workunit.client.0.trial112.stdout: "last_reqid": "client.4252.0:1", 2026-02-13T02:27:50.729 INFO:tasks.workunit.client.0.trial112.stdout: "user_version": 38, 2026-02-13T02:27:50.729 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:27:50.729 INFO:tasks.workunit.client.0.trial112.stdout: "mtime": "2026-02-13T02:27:26.930448+0000", 2026-02-13T02:27:50.729 INFO:tasks.workunit.client.0.trial112.stdout: "local_mtime": "2026-02-13T02:27:26.930730+0000", 2026-02-13T02:27:50.729 INFO:tasks.workunit.client.0.trial112.stdout: "lost": 0, 2026-02-13T02:27:50.729 INFO:tasks.workunit.client.0.trial112.stdout: "flags": [ 2026-02-13T02:27:50.729 INFO:tasks.workunit.client.0.trial112.stdout: "dirty", 2026-02-13T02:27:50.729 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest" 2026-02-13T02:27:50.729 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:27:50.729 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_seq": 0, 2026-02-13T02:27:50.729 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_size": 0, 2026-02-13T02:27:50.729 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest": "0xc6ceb5ce", 2026-02-13T02:27:50.729 INFO:tasks.workunit.client.0.trial112.stdout: "omap_digest": "0xffffffff", 2026-02-13T02:27:50.729 INFO:tasks.workunit.client.0.trial112.stdout: "expected_object_size": 0, 2026-02-13T02:27:50.729 INFO:tasks.workunit.client.0.trial112.stdout: "expected_write_size": 0, 2026-02-13T02:27:50.729 INFO:tasks.workunit.client.0.trial112.stdout: "alloc_hint_flags": 0, 2026-02-13T02:27:50.729 INFO:tasks.workunit.client.0.trial112.stdout: "manifest": { 2026-02-13T02:27:50.729 INFO:tasks.workunit.client.0.trial112.stdout: "type": 0 2026-02-13T02:27:50.729 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.729 INFO:tasks.workunit.client.0.trial112.stdout: "watchers": {}, 2026-02-13T02:27:50.729 INFO:tasks.workunit.client.0.trial112.stdout: "shard_versions": [] 2026-02-13T02:27:50.729 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.729 INFO:tasks.workunit.client.0.trial112.stdout: "shards": [ 2026-02-13T02:27:50.729 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.729 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 0, 2026-02-13T02:27:50.729 INFO:tasks.workunit.client.0.trial112.stdout: "primary": false, 2026-02-13T02:27:50.729 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:27:50.729 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:27:50.729 INFO:tasks.workunit.client.0.trial112.stdout: "snapset": { 2026-02-13T02:27:50.729 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 1, 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: "clones": [ 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: "snap": 1, 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: "size": 1032, 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: "overlap": "[]", 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: "snaps": [ 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: 1 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 1, 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: "primary": true, 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: "snapset": { 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 1, 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: "clones": [ 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: "snap": 1, 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: "size": 1032, 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: "overlap": "[]" 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: "object": { 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: "name": "obj14", 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: "nspace": "", 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: "locator": "", 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: "snap": "head", 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: "version": 40 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [ 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: "snapset_inconsistency" 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:27:50.730 INFO:tasks.workunit.client.0.trial112.stdout: "union_shard_errors": [], 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: "selected_object_info": { 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: "oid": { 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: "oid": "obj14", 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: "key": "", 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: "snapid": -2, 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: "hash": 2484217095, 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: "max": 0, 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: "pool": 1, 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: "namespace": "" 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: "version": "16'40", 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: "prior_version": "15'14", 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: "last_reqid": "client.4255.0:1", 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: "user_version": 40, 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: "mtime": "2026-02-13T02:27:26.958465+0000", 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: "local_mtime": "2026-02-13T02:27:26.958829+0000", 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: "lost": 0, 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: "flags": [ 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: "dirty", 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest" 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_seq": 0, 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_size": 0, 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest": "0xc6ceb5ce", 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: "omap_digest": "0xffffffff", 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: "expected_object_size": 0, 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: "expected_write_size": 0, 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: "alloc_hint_flags": 0, 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: "manifest": { 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: "type": 0 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: "watchers": {}, 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: "shard_versions": [] 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: "shards": [ 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 0, 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: "primary": false, 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:27:50.731 INFO:tasks.workunit.client.0.trial112.stdout: "snapset": { 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 1, 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: "clones": [ 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: "snap": 1, 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: "size": 1032, 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: "overlap": "[]", 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: "snaps": [ 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: 1 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 1, 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: "primary": true, 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: "snapset": { 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 1, 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: "clones": [ 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: "snap": 1, 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: "size": 1033, 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: "overlap": "[]", 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: "snaps": [ 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: 1 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: "object": { 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: "name": "obj15", 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: "nspace": "", 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: "locator": "", 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: "snap": "head", 2026-02-13T02:27:50.732 INFO:tasks.workunit.client.0.trial112.stdout: "version": 15 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: "union_shard_errors": [ 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: "snapset_corrupted" 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: "selected_object_info": { 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: "oid": { 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: "oid": "obj15", 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: "key": "", 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: "snapid": -2, 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: "hash": 612772309, 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: "max": 0, 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: "pool": 1, 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: "namespace": "" 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: "version": "15'15", 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: "prior_version": "0'0", 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: "last_reqid": "client.4207.0:1", 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: "user_version": 15, 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: "size": 1032, 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: "mtime": "2026-02-13T02:27:26.009391+0000", 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: "local_mtime": "2026-02-13T02:27:26.009735+0000", 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: "lost": 0, 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: "flags": [ 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: "dirty", 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest" 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_seq": 0, 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_size": 0, 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest": "0x10ab1204", 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: "omap_digest": "0xffffffff", 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: "expected_object_size": 0, 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: "expected_write_size": 0, 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: "alloc_hint_flags": 0, 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: "manifest": { 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: "type": 0 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: "watchers": {}, 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: "shard_versions": [] 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: "shards": [ 2026-02-13T02:27:50.733 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 0, 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: "primary": false, 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: "size": 1032, 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: "snapset": { 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 0, 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: "clones": [] 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 1, 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: "primary": true, 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [ 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: "snapset_corrupted" 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: "size": 1032, 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: "snapset": "Z2FyYmFnZQo=" 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: "object": { 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: "name": "obj2", 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: "nspace": "", 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: "locator": "", 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: "snap": "head", 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: "version": 56 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: "union_shard_errors": [ 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: "snapset_missing" 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: "selected_object_info": { 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: "oid": { 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: "oid": "obj2", 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: "key": "", 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: "snapid": -2, 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: "hash": 1058988552, 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: "max": 0, 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: "pool": 1, 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: "namespace": "" 2026-02-13T02:27:50.734 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: "version": "22'56", 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: "prior_version": "19'48", 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: "last_reqid": "client.4297.0:1", 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: "user_version": 56, 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: "size": 0, 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: "mtime": "2026-02-13T02:27:27.620372+0000", 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: "local_mtime": "2026-02-13T02:27:27.620770+0000", 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: "lost": 0, 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: "flags": [ 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: "whiteout", 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: "dirty" 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_seq": 0, 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_size": 0, 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest": "0xffffffff", 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: "omap_digest": "0xffffffff", 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: "expected_object_size": 0, 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: "expected_write_size": 0, 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: "alloc_hint_flags": 0, 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: "manifest": { 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: "type": 0 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: "watchers": {}, 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: "shard_versions": [] 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: "shards": [ 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 0, 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: "primary": false, 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: "size": 0, 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: "snapset": { 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 7, 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: "clones": [ 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: "snap": 4, 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: "size": 1032, 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: "overlap": "[]", 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: "snaps": [ 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: 4, 2026-02-13T02:27:50.735 INFO:tasks.workunit.client.0.trial112.stdout: 3, 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: 2, 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: 1 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: "snap": 7, 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: "size": 1024, 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: "overlap": "[]", 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: "snaps": [ 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: 7, 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: 6, 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: 5 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 1, 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: "primary": true, 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [ 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: "snapset_missing" 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: "size": 0 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: "object": { 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: "name": "obj3", 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: "nspace": "", 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: "locator": "", 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: "snap": "head", 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: "version": 44 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [ 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: "size_mismatch" 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: "union_shard_errors": [ 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: "size_mismatch_info", 2026-02-13T02:27:50.736 INFO:tasks.workunit.client.0.trial112.stdout: "obj_size_info_mismatch" 2026-02-13T02:27:50.737 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:27:50.737 INFO:tasks.workunit.client.0.trial112.stdout: "selected_object_info": { 2026-02-13T02:27:50.737 INFO:tasks.workunit.client.0.trial112.stdout: "oid": { 2026-02-13T02:27:50.737 INFO:tasks.workunit.client.0.trial112.stdout: "oid": "obj3", 2026-02-13T02:27:50.737 INFO:tasks.workunit.client.0.trial112.stdout: "key": "", 2026-02-13T02:27:50.737 INFO:tasks.workunit.client.0.trial112.stdout: "snapid": -2, 2026-02-13T02:27:50.737 INFO:tasks.workunit.client.0.trial112.stdout: "hash": 1643547569, 2026-02-13T02:27:50.737 INFO:tasks.workunit.client.0.trial112.stdout: "max": 0, 2026-02-13T02:27:50.737 INFO:tasks.workunit.client.0.trial112.stdout: "pool": 1, 2026-02-13T02:27:50.737 INFO:tasks.workunit.client.0.trial112.stdout: "namespace": "" 2026-02-13T02:27:50.737 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.737 INFO:tasks.workunit.client.0.trial112.stdout: "version": "18'44", 2026-02-13T02:27:50.737 INFO:tasks.workunit.client.0.trial112.stdout: "prior_version": "16'22", 2026-02-13T02:27:50.737 INFO:tasks.workunit.client.0.trial112.stdout: "last_reqid": "client.4267.0:1", 2026-02-13T02:27:50.737 INFO:tasks.workunit.client.0.trial112.stdout: "user_version": 44, 2026-02-13T02:27:50.737 INFO:tasks.workunit.client.0.trial112.stdout: "size": 768, 2026-02-13T02:27:50.737 INFO:tasks.workunit.client.0.trial112.stdout: "mtime": "2026-02-13T02:27:27.169406+0000", 2026-02-13T02:27:50.737 INFO:tasks.workunit.client.0.trial112.stdout: "local_mtime": "2026-02-13T02:27:27.169702+0000", 2026-02-13T02:27:50.737 INFO:tasks.workunit.client.0.trial112.stdout: "lost": 0, 2026-02-13T02:27:50.737 INFO:tasks.workunit.client.0.trial112.stdout: "flags": [ 2026-02-13T02:27:50.737 INFO:tasks.workunit.client.0.trial112.stdout: "dirty", 2026-02-13T02:27:50.737 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest" 2026-02-13T02:27:50.737 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:27:50.737 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_seq": 0, 2026-02-13T02:27:50.737 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_size": 0, 2026-02-13T02:27:50.737 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest": "0x31862c90", 2026-02-13T02:27:50.737 INFO:tasks.workunit.client.0.trial112.stdout: "omap_digest": "0xffffffff", 2026-02-13T02:27:50.737 INFO:tasks.workunit.client.0.trial112.stdout: "expected_object_size": 0, 2026-02-13T02:27:50.737 INFO:tasks.workunit.client.0.trial112.stdout: "expected_write_size": 0, 2026-02-13T02:27:50.737 INFO:tasks.workunit.client.0.trial112.stdout: "alloc_hint_flags": 0, 2026-02-13T02:27:50.737 INFO:tasks.workunit.client.0.trial112.stdout: "manifest": { 2026-02-13T02:27:50.737 INFO:tasks.workunit.client.0.trial112.stdout: "type": 0 2026-02-13T02:27:50.737 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.737 INFO:tasks.workunit.client.0.trial112.stdout: "watchers": {}, 2026-02-13T02:27:50.737 INFO:tasks.workunit.client.0.trial112.stdout: "shard_versions": [] 2026-02-13T02:27:50.737 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.737 INFO:tasks.workunit.client.0.trial112.stdout: "shards": [ 2026-02-13T02:27:50.737 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.737 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 0, 2026-02-13T02:27:50.737 INFO:tasks.workunit.client.0.trial112.stdout: "primary": false, 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: "size": 768 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 1, 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: "primary": true, 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [ 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: "size_mismatch_info", 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: "obj_size_info_mismatch" 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: "size": 3840, 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: "object_info": { 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: "oid": { 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: "oid": "obj3", 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: "key": "", 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: "snapid": -2, 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: "hash": 1643547569, 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: "max": 0, 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: "pool": 1, 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: "namespace": "" 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: "version": "18'44", 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: "prior_version": "16'22", 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: "last_reqid": "client.4267.0:1", 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: "user_version": 44, 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: "size": 768, 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: "mtime": "2026-02-13T02:27:27.169406+0000", 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: "local_mtime": "2026-02-13T02:27:27.169702+0000", 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: "lost": 0, 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: "flags": [ 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: "dirty", 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest" 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_seq": 0, 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_size": 0, 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest": "0x31862c90", 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: "omap_digest": "0xffffffff", 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: "expected_object_size": 0, 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: "expected_write_size": 0, 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: "alloc_hint_flags": 0, 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: "manifest": { 2026-02-13T02:27:50.738 INFO:tasks.workunit.client.0.trial112.stdout: "type": 0 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: "watchers": {}, 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: "shard_versions": [] 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: "object": { 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: "name": "obj6", 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: "nspace": "", 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: "locator": "", 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: "snap": "head", 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: "version": 24 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [ 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: "snapset_inconsistency" 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: "union_shard_errors": [], 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: "selected_object_info": { 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: "oid": { 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: "oid": "obj6", 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: "key": "", 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: "snapid": -2, 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: "hash": 2202164420, 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: "max": 0, 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: "pool": 1, 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: "namespace": "" 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: "version": "16'24", 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: "prior_version": "15'6", 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: "last_reqid": "client.4231.0:1", 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: "user_version": 24, 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: "mtime": "2026-02-13T02:27:26.729560+0000", 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: "local_mtime": "2026-02-13T02:27:26.729928+0000", 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: "lost": 0, 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: "flags": [ 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: "dirty", 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest" 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_seq": 0, 2026-02-13T02:27:50.739 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_size": 0, 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest": "0xc6ceb5ce", 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: "omap_digest": "0xffffffff", 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: "expected_object_size": 0, 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: "expected_write_size": 0, 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: "alloc_hint_flags": 0, 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: "manifest": { 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: "type": 0 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: "watchers": {}, 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: "shard_versions": [] 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: "shards": [ 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 0, 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: "primary": false, 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: "snapset": { 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 1, 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: "clones": [ 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: "snap": 1, 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: "size": 1032, 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: "overlap": "[]", 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: "snaps": [ 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: 1 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 1, 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: "primary": true, 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: "snapset": { 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 1, 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: "clones": [] 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.740 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.741 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.741 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.741 INFO:tasks.workunit.client.0.trial112.stdout: "object": { 2026-02-13T02:27:50.741 INFO:tasks.workunit.client.0.trial112.stdout: "name": "obj7", 2026-02-13T02:27:50.741 INFO:tasks.workunit.client.0.trial112.stdout: "nspace": "", 2026-02-13T02:27:50.741 INFO:tasks.workunit.client.0.trial112.stdout: "locator": "", 2026-02-13T02:27:50.741 INFO:tasks.workunit.client.0.trial112.stdout: "snap": "head", 2026-02-13T02:27:50.741 INFO:tasks.workunit.client.0.trial112.stdout: "version": 26 2026-02-13T02:27:50.741 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.741 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [ 2026-02-13T02:27:50.741 INFO:tasks.workunit.client.0.trial112.stdout: "snapset_inconsistency" 2026-02-13T02:27:50.741 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:27:50.741 INFO:tasks.workunit.client.0.trial112.stdout: "union_shard_errors": [], 2026-02-13T02:27:50.741 INFO:tasks.workunit.client.0.trial112.stdout: "selected_object_info": { 2026-02-13T02:27:50.741 INFO:tasks.workunit.client.0.trial112.stdout: "oid": { 2026-02-13T02:27:50.741 INFO:tasks.workunit.client.0.trial112.stdout: "oid": "obj7", 2026-02-13T02:27:50.741 INFO:tasks.workunit.client.0.trial112.stdout: "key": "", 2026-02-13T02:27:50.741 INFO:tasks.workunit.client.0.trial112.stdout: "snapid": -2, 2026-02-13T02:27:50.741 INFO:tasks.workunit.client.0.trial112.stdout: "hash": 1552453721, 2026-02-13T02:27:50.741 INFO:tasks.workunit.client.0.trial112.stdout: "max": 0, 2026-02-13T02:27:50.741 INFO:tasks.workunit.client.0.trial112.stdout: "pool": 1, 2026-02-13T02:27:50.741 INFO:tasks.workunit.client.0.trial112.stdout: "namespace": "" 2026-02-13T02:27:50.741 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.741 INFO:tasks.workunit.client.0.trial112.stdout: "version": "16'26", 2026-02-13T02:27:50.741 INFO:tasks.workunit.client.0.trial112.stdout: "prior_version": "15'7", 2026-02-13T02:27:50.741 INFO:tasks.workunit.client.0.trial112.stdout: "last_reqid": "client.4234.0:1", 2026-02-13T02:27:50.741 INFO:tasks.workunit.client.0.trial112.stdout: "user_version": 26, 2026-02-13T02:27:50.741 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:27:50.741 INFO:tasks.workunit.client.0.trial112.stdout: "mtime": "2026-02-13T02:27:26.757994+0000", 2026-02-13T02:27:50.741 INFO:tasks.workunit.client.0.trial112.stdout: "local_mtime": "2026-02-13T02:27:26.758371+0000", 2026-02-13T02:27:50.741 INFO:tasks.workunit.client.0.trial112.stdout: "lost": 0, 2026-02-13T02:27:50.741 INFO:tasks.workunit.client.0.trial112.stdout: "flags": [ 2026-02-13T02:27:50.741 INFO:tasks.workunit.client.0.trial112.stdout: "dirty", 2026-02-13T02:27:50.741 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest" 2026-02-13T02:27:50.741 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:27:50.741 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_seq": 0, 2026-02-13T02:27:50.741 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_size": 0, 2026-02-13T02:27:50.741 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest": "0xc6ceb5ce", 2026-02-13T02:27:50.741 INFO:tasks.workunit.client.0.trial112.stdout: "omap_digest": "0xffffffff", 2026-02-13T02:27:50.741 INFO:tasks.workunit.client.0.trial112.stdout: "expected_object_size": 0, 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: "expected_write_size": 0, 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: "alloc_hint_flags": 0, 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: "manifest": { 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: "type": 0 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: "watchers": {}, 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: "shard_versions": [] 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: "shards": [ 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 0, 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: "primary": false, 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: "snapset": { 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 1, 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: "clones": [ 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: "snap": 1, 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: "size": 1032, 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: "overlap": "[]", 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: "snaps": [ 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: 1 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 1, 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: "primary": true, 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: "snapset": { 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 0, 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: "clones": [] 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.742 INFO:tasks.workunit.client.0.trial112.stdout: "object": { 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: "name": "obj8", 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: "nspace": "", 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: "locator": "", 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: "snap": "head", 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: "version": 28 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [ 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: "snapset_inconsistency" 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: "union_shard_errors": [], 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: "selected_object_info": { 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: "oid": { 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: "oid": "obj8", 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: "key": "", 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: "snapid": -2, 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: "hash": 2381834917, 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: "max": 0, 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: "pool": 1, 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: "namespace": "" 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: "version": "16'28", 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: "prior_version": "15'8", 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: "last_reqid": "client.4237.0:1", 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: "user_version": 28, 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: "mtime": "2026-02-13T02:27:26.786717+0000", 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: "local_mtime": "2026-02-13T02:27:26.787065+0000", 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: "lost": 0, 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: "flags": [ 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: "dirty", 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest" 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_seq": 0, 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_size": 0, 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest": "0xc6ceb5ce", 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: "omap_digest": "0xffffffff", 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: "expected_object_size": 0, 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: "expected_write_size": 0, 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: "alloc_hint_flags": 0, 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: "manifest": { 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: "type": 0 2026-02-13T02:27:50.743 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.744 INFO:tasks.workunit.client.0.trial112.stdout: "watchers": {}, 2026-02-13T02:27:50.744 INFO:tasks.workunit.client.0.trial112.stdout: "shard_versions": [] 2026-02-13T02:27:50.744 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.744 INFO:tasks.workunit.client.0.trial112.stdout: "shards": [ 2026-02-13T02:27:50.744 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.744 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 0, 2026-02-13T02:27:50.744 INFO:tasks.workunit.client.0.trial112.stdout: "primary": false, 2026-02-13T02:27:50.744 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:27:50.744 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:27:50.744 INFO:tasks.workunit.client.0.trial112.stdout: "snapset": { 2026-02-13T02:27:50.744 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 1, 2026-02-13T02:27:50.744 INFO:tasks.workunit.client.0.trial112.stdout: "clones": [ 2026-02-13T02:27:50.744 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.744 INFO:tasks.workunit.client.0.trial112.stdout: "snap": 1, 2026-02-13T02:27:50.744 INFO:tasks.workunit.client.0.trial112.stdout: "size": 1032, 2026-02-13T02:27:50.744 INFO:tasks.workunit.client.0.trial112.stdout: "overlap": "[]", 2026-02-13T02:27:50.744 INFO:tasks.workunit.client.0.trial112.stdout: "snaps": [ 2026-02-13T02:27:50.744 INFO:tasks.workunit.client.0.trial112.stdout: 1 2026-02-13T02:27:50.744 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.744 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.744 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.744 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.744 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.744 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.744 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 1, 2026-02-13T02:27:50.744 INFO:tasks.workunit.client.0.trial112.stdout: "primary": true, 2026-02-13T02:27:50.744 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:27:50.744 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:27:50.744 INFO:tasks.workunit.client.0.trial112.stdout: "snapset": { 2026-02-13T02:27:50.744 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 0, 2026-02-13T02:27:50.744 INFO:tasks.workunit.client.0.trial112.stdout: "clones": [ 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: "snap": 1, 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: "size": 1032, 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: "overlap": "[]", 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: "snaps": [ 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: 1 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: "object": { 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: "name": "obj9", 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: "nspace": "", 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: "locator": "", 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: "snap": "head", 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: "version": 30 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [ 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: "snapset_inconsistency" 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: "union_shard_errors": [], 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: "selected_object_info": { 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: "oid": { 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: "oid": "obj9", 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: "key": "", 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: "snapid": -2, 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: "hash": 3833113727, 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: "max": 0, 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: "pool": 1, 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: "namespace": "" 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: "version": "16'30", 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: "prior_version": "15'9", 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: "last_reqid": "client.4240.0:1", 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: "user_version": 30, 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:27:50.745 INFO:tasks.workunit.client.0.trial112.stdout: "mtime": "2026-02-13T02:27:26.816615+0000", 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: "local_mtime": "2026-02-13T02:27:26.816938+0000", 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: "lost": 0, 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: "flags": [ 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: "dirty", 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest" 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_seq": 0, 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_size": 0, 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest": "0xc6ceb5ce", 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: "omap_digest": "0xffffffff", 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: "expected_object_size": 0, 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: "expected_write_size": 0, 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: "alloc_hint_flags": 0, 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: "manifest": { 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: "type": 0 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: "watchers": {}, 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: "shard_versions": [] 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: "shards": [ 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 0, 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: "primary": false, 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: "snapset": { 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 1, 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: "clones": [ 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: "snap": 1, 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: "size": 1032, 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: "overlap": "[]", 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: "snaps": [ 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: 1 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.746 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 1, 2026-02-13T02:27:50.747 INFO:tasks.workunit.client.0.trial112.stdout: "primary": true, 2026-02-13T02:27:50.747 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:27:50.747 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:27:50.747 INFO:tasks.workunit.client.0.trial112.stdout: "snapset": { 2026-02-13T02:27:50.747 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 1, 2026-02-13T02:27:50.747 INFO:tasks.workunit.client.0.trial112.stdout: "clones": [ 2026-02-13T02:27:50.747 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:27:50.747 INFO:tasks.workunit.client.0.trial112.stdout: "snap": 1, 2026-02-13T02:27:50.747 INFO:tasks.workunit.client.0.trial112.stdout: "size": "????", 2026-02-13T02:27:50.747 INFO:tasks.workunit.client.0.trial112.stdout: "overlap": "[]", 2026-02-13T02:27:50.747 INFO:tasks.workunit.client.0.trial112.stdout: "snaps": [ 2026-02-13T02:27:50.747 INFO:tasks.workunit.client.0.trial112.stdout: 1 2026-02-13T02:27:50.747 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.747 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.747 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.747 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.747 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.747 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.747 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:27:50.747 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:27:50.747 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:27:50.747 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:797: _scrub_snaps_multi: rados list-inconsistent-snapset 1.0 2026-02-13T02:27:50.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:800: _scrub_snaps_multi: '[' primary = replica ']' 2026-02-13T02:27:50.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:811: _scrub_snaps_multi: scruberrors=30 2026-02-13T02:27:50.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:812: _scrub_snaps_multi: jq .inconsistents 2026-02-13T02:27:50.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:812: _scrub_snaps_multi: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2026-02-13T02:27:50.768 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1066: _scrub_snaps_multi: jq .inconsistents td/osd-scrub-snaps/json 2026-02-13T02:27:50.768 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1066: _scrub_snaps_multi: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2026-02-13T02:27:50.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1067: _scrub_snaps_multi: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2026-02-13T02:27:50.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2513: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2026-02-13T02:27:50.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1068: _scrub_snaps_multi: test no = yes 2026-02-13T02:27:50.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1073: _scrub_snaps_multi: test '' = yes 2026-02-13T02:27:50.784 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: find td/osd-scrub-snaps 2026-02-13T02:27:50.784 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: grep 'osd[^/]*\.pid' 2026-02-13T02:27:50.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: pidfiles='td/osd-scrub-snaps/osd.1.pid 2026-02-13T02:27:50.786 INFO:tasks.workunit.client.0.trial112.stderr:td/osd-scrub-snaps/osd.0.pid' 2026-02-13T02:27:50.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1079: _scrub_snaps_multi: pids= 2026-02-13T02:27:50.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1080: _scrub_snaps_multi: for pidfile in ${pidfiles} 2026-02-13T02:27:50.787 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: cat td/osd-scrub-snaps/osd.1.pid 2026-02-13T02:27:50.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: pids+='487743 ' 2026-02-13T02:27:50.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1080: _scrub_snaps_multi: for pidfile in ${pidfiles} 2026-02-13T02:27:50.788 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: cat td/osd-scrub-snaps/osd.0.pid 2026-02-13T02:27:50.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: pids+='487201 ' 2026-02-13T02:27:50.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1085: _scrub_snaps_multi: ERRORS=0 2026-02-13T02:27:50.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1089: _scrub_snaps_multi: '[' primary = primary ']' 2026-02-13T02:27:50.789 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: seq 1 7 2026-02-13T02:27:50.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2026-02-13T02:27:50.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap1 2026-02-13T02:27:50.862 INFO:tasks.workunit.client.0.trial112.stdout:removed pool test snap snap1 2026-02-13T02:27:50.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2026-02-13T02:27:50.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap2 2026-02-13T02:27:50.964 INFO:tasks.workunit.client.0.trial112.stdout:removed pool test snap snap2 2026-02-13T02:27:50.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2026-02-13T02:27:50.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap3 2026-02-13T02:27:51.067 INFO:tasks.workunit.client.0.trial112.stdout:removed pool test snap snap3 2026-02-13T02:27:51.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2026-02-13T02:27:51.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap4 2026-02-13T02:27:51.170 INFO:tasks.workunit.client.0.trial112.stdout:removed pool test snap snap4 2026-02-13T02:27:51.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2026-02-13T02:27:51.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap5 2026-02-13T02:27:51.272 INFO:tasks.workunit.client.0.trial112.stdout:removed pool test snap snap5 2026-02-13T02:27:51.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2026-02-13T02:27:51.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap6 2026-02-13T02:27:51.374 INFO:tasks.workunit.client.0.trial112.stdout:removed pool test snap snap6 2026-02-13T02:27:51.376 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2026-02-13T02:27:51.376 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap7 2026-02-13T02:27:51.476 INFO:tasks.workunit.client.0.trial112.stdout:removed pool test snap snap7 2026-02-13T02:27:51.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1095: _scrub_snaps_multi: sleep 5 2026-02-13T02:27:56.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1096: _scrub_snaps_multi: local -i loop=0 2026-02-13T02:27:56.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1097: _scrub_snaps_multi: ceph pg dump pgs 2026-02-13T02:27:56.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1097: _scrub_snaps_multi: grep -q snaptrim 2026-02-13T02:27:56.628 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:27:56.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1099: _scrub_snaps_multi: ceph pg dump pgs 2026-02-13T02:27:56.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1099: _scrub_snaps_multi: grep -q snaptrim_error 2026-02-13T02:27:56.805 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:27:56.816 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1101: _scrub_snaps_multi: break 2026-02-13T02:27:56.816 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1111: _scrub_snaps_multi: ceph pg dump pgs 2026-02-13T02:27:56.967 INFO:tasks.workunit.client.0.trial112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-13T02:27:56.967 INFO:tasks.workunit.client.0.trial112.stdout:1.0 36 0 0 0 0 24448 0 0 56 0 56 active+clean+inconsistent+snaptrim_error 2026-02-13T02:27:51.470459+0000 22'56 37:132 [1,0] 1 [1,0] 1 22'56 2026-02-13T02:27:46.733768+0000 0'0 2026-02-13T02:27:21.891155+0000 7 1 periodic scrub scheduled @ 2026-02-14T11:29:16.666427+0000 33 0 2026-02-13T02:27:56.967 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T02:27:56.968 INFO:tasks.workunit.client.0.trial112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-13T02:27:56.968 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:27:56.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1113: _scrub_snaps_multi: for pid in $pids 2026-02-13T02:27:56.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1115: _scrub_snaps_multi: kill -0 487743 2026-02-13T02:27:56.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1113: _scrub_snaps_multi: for pid in $pids 2026-02-13T02:27:56.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1115: _scrub_snaps_multi: kill -0 487201 2026-02-13T02:27:56.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1122: _scrub_snaps_multi: kill_daemons td/osd-scrub-snaps 2026-02-13T02:27:56.977 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:27:56.977 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:27:56.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:27:56.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:27:56.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:28:02.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:28:02.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1124: _scrub_snaps_multi: declare -a err_strings 2026-02-13T02:28:02.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1125: _scrub_snaps_multi: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj4:7 : missing' 2026-02-13T02:28:02.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1126: _scrub_snaps_multi: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj3:head : size 3840 != size 768 from auth oi' 2026-02-13T02:28:02.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1127: _scrub_snaps_multi: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:1 : missing' 2026-02-13T02:28:02.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1128: _scrub_snaps_multi: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:2 : missing' 2026-02-13T02:28:02.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1129: _scrub_snaps_multi: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj5:4 : size 4608 != size 512 from auth oi' 2026-02-13T02:28:02.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1130: _scrub_snaps_multi: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid .*:::obj5:7 : failed to pick suitable object info' 2026-02-13T02:28:02.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj1:head : missing' 2026-02-13T02:28:02.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1132: _scrub_snaps_multi: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 30 errors' 2026-02-13T02:28:02.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-13T02:28:02.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj4:7 : missing' td/osd-scrub-snaps/osd.1.log 2026-02-13T02:28:02.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-13T02:28:02.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj3:head : size 3840 != size 768 from auth oi' td/osd-scrub-snaps/osd.1.log 2026-02-13T02:28:02.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-13T02:28:02.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:1 : missing' td/osd-scrub-snaps/osd.1.log 2026-02-13T02:28:02.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-13T02:28:02.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:2 : missing' td/osd-scrub-snaps/osd.1.log 2026-02-13T02:28:02.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-13T02:28:02.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj5:4 : size 4608 != size 512 from auth oi' td/osd-scrub-snaps/osd.1.log 2026-02-13T02:28:02.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-13T02:28:02.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid .*:::obj5:7 : failed to pick suitable object info' td/osd-scrub-snaps/osd.1.log 2026-02-13T02:28:02.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-13T02:28:02.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj1:head : missing' td/osd-scrub-snaps/osd.1.log 2026-02-13T02:28:02.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-13T02:28:02.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 30 errors' td/osd-scrub-snaps/osd.1.log 2026-02-13T02:28:02.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1144: _scrub_snaps_multi: declare -a rep_err_strings 2026-02-13T02:28:02.313 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: eval echo '$primary' 2026-02-13T02:28:02.313 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: echo 1 2026-02-13T02:28:02.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: osd=1 2026-02-13T02:28:02.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1146: _scrub_snaps_multi: rep_err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' 2026-02-13T02:28:02.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1147: _scrub_snaps_multi: for err_string in "${rep_err_strings[@]}" 2026-02-13T02:28:02.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1149: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' td/osd-scrub-snaps/osd.1.log 2026-02-13T02:28:02.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1156: _scrub_snaps_multi: '[' 0 '!=' 0 ']' 2026-02-13T02:28:02.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1162: _scrub_snaps_multi: echo 'TEST PASSED' 2026-02-13T02:28:02.317 INFO:tasks.workunit.client.0.trial112.stdout:TEST PASSED 2026-02-13T02:28:02.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1163: _scrub_snaps_multi: return 0 2026-02-13T02:28:02.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1181: TEST_scrub_snaps_primary: err=0 2026-02-13T02:28:02.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1182: TEST_scrub_snaps_primary: CEPH_ARGS='--fsid=b2b95964-f5a3-4b20-bd2b-d60f1ae5f8a0 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2026-02-13T02:28:02.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1183: TEST_scrub_snaps_primary: return 0 2026-02-13T02:28:02.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:40: run: teardown td/osd-scrub-snaps 2026-02-13T02:28:02.319 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2026-02-13T02:28:02.319 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:28:02.319 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2026-02-13T02:28:02.319 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:28:02.319 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:28:02.319 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:28:02.319 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:28:02.319 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:28:02.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:28:02.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:28:02.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:28:02.325 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:28:02.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:28:02.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:28:02.326 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:28:02.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:28:02.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:28:02.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:28:02.328 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:28:02.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:28:02.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:28:02.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2026-02-13T02:28:02.342 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:28:02.342 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:28:02.342 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.453125 2026-02-13T02:28:02.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.453125 2026-02-13T02:28:02.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:28:02.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:28:02.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:37: run: for func in $funcs 2026-02-13T02:28:02.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:38: run: setup td/osd-scrub-snaps 2026-02-13T02:28:02.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-snaps 2026-02-13T02:28:02.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-snaps 2026-02-13T02:28:02.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2026-02-13T02:28:02.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:28:02.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2026-02-13T02:28:02.345 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:28:02.346 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:28:02.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:28:02.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:28:02.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:28:02.347 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:28:02.348 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:28:02.348 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:28:02.349 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:28:02.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:28:02.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:28:02.351 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:28:02.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:28:02.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:28:02.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:28:02.353 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:28:02.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:28:02.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:28:02.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2026-02-13T02:28:02.356 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:28:02.356 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:28:02.356 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.453125 2026-02-13T02:28:02.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.453125 2026-02-13T02:28:02.357 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:28:02.357 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:28:02.357 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-snaps 2026-02-13T02:28:02.358 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T02:28:02.359 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:28:02.359 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.453125 2026-02-13T02:28:02.359 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.453125 2026-02-13T02:28:02.360 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T02:28:02.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-13T02:28:02.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T02:28:02.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-snaps 1' TERM HUP INT 2026-02-13T02:28:02.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:39: run: TEST_scrub_snaps_replica td/osd-scrub-snaps 2026-02-13T02:28:02.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1167: TEST_scrub_snaps_replica: local dir=td/osd-scrub-snaps 2026-02-13T02:28:02.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1168: TEST_scrub_snaps_replica: ORIG_ARGS='--fsid=b2b95964-f5a3-4b20-bd2b-d60f1ae5f8a0 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2026-02-13T02:28:02.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1169: TEST_scrub_snaps_replica: CEPH_ARGS+=' --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2026-02-13T02:28:02.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1170: TEST_scrub_snaps_replica: _scrub_snaps_multi td/osd-scrub-snaps replica 2026-02-13T02:28:02.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:732: _scrub_snaps_multi: local dir=td/osd-scrub-snaps 2026-02-13T02:28:02.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:733: _scrub_snaps_multi: local poolname=test 2026-02-13T02:28:02.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:734: _scrub_snaps_multi: local OBJS=16 2026-02-13T02:28:02.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:735: _scrub_snaps_multi: local OSDS=2 2026-02-13T02:28:02.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:736: _scrub_snaps_multi: local which=replica 2026-02-13T02:28:02.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:738: _scrub_snaps_multi: TESTDATA=testdata.453125 2026-02-13T02:28:02.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:740: _scrub_snaps_multi: run_mon td/osd-scrub-snaps a --osd_pool_default_size=2 2026-02-13T02:28:02.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-snaps 2026-02-13T02:28:02.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-13T02:28:02.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-13T02:28:02.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-13T02:28:02.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-snaps/a 2026-02-13T02:28:02.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-snaps/a --run-dir=td/osd-scrub-snaps --osd_pool_default_size=2 2026-02-13T02:28:02.387 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-13T02:28:02.388 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:28:02.388 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:28:02.388 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:28:02.388 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:28:02.388 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.453125 2026-02-13T02:28:02.388 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.453125/$cluster-$name.asok' 2026-02-13T02:28:02.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-snaps/a '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.453125/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-snaps/log --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2026-02-13T02:28:02.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-13T02:28:02.420 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-13T02:28:02.420 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:28:02.420 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:28:02.420 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-13T02:28:02.421 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-13T02:28:02.422 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:28:02.422 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:28:02.422 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:28:02.422 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:28:02.422 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:28:02.422 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.453125 2026-02-13T02:28:02.422 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.453125/ceph-mon.a.asok 2026-02-13T02:28:02.422 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:28:02.422 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.453125/ceph-mon.a.asok config get fsid 2026-02-13T02:28:02.496 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-13T02:28:02.496 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:28:02.497 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:28:02.497 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-13T02:28:02.497 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-13T02:28:02.498 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:28:02.498 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:28:02.498 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:28:02.498 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:28:02.498 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:28:02.499 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.453125 2026-02-13T02:28:02.499 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.453125/ceph-mon.a.asok 2026-02-13T02:28:02.499 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:28:02.499 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.453125/ceph-mon.a.asok config get mon_host 2026-02-13T02:28:02.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:741: _scrub_snaps_multi: run_mgr td/osd-scrub-snaps x 2026-02-13T02:28:02.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-snaps 2026-02-13T02:28:02.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-13T02:28:02.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-13T02:28:02.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-13T02:28:02.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-snaps/x 2026-02-13T02:28:02.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-13T02:28:02.655 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-13T02:28:02.656 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:28:02.656 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:28:02.656 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:28:02.656 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:28:02.656 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.453125 2026-02-13T02:28:02.656 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.453125/$cluster-$name.asok' 2026-02-13T02:28:02.657 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:28:02.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-snaps/x '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.453125/$cluster-$name.asok' --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:28:02.679 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: expr 2 - 1 2026-02-13T02:28:02.681 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: seq 0 1 2026-02-13T02:28:02.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:28:02.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:744: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 0 2026-02-13T02:28:02.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-snaps 2026-02-13T02:28:02.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:28:02.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-13T02:28:02.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:28:02.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-snaps/0 2026-02-13T02:28:02.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=b2b95964-f5a3-4b20-bd2b-d60f1ae5f8a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2026-02-13T02:28:02.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:28:02.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:28:02.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:28:02.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2026-02-13T02:28:02.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2026-02-13T02:28:02.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:28:02.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:28:02.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2026-02-13T02:28:02.686 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:28:02.686 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:28:02.686 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:28:02.686 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:28:02.686 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:28:02.686 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.453125 2026-02-13T02:28:02.686 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.453125/$cluster-$name.asok' 2026-02-13T02:28:02.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.453125/$cluster-$name.asok' 2026-02-13T02:28:02.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:28:02.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:28:02.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:28:02.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2026-02-13T02:28:02.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2026-02-13T02:28:02.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:28:02.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:28:02.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:28:02.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:28:02.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:28:02.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:28:02.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-snaps/0 2026-02-13T02:28:02.688 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:28:02.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4ad2bdde-fc58-4696-9ca8-10db52bfba32 2026-02-13T02:28:02.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 4ad2bdde-fc58-4696-9ca8-10db52bfba32' 2026-02-13T02:28:02.688 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 4ad2bdde-fc58-4696-9ca8-10db52bfba32 2026-02-13T02:28:02.689 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:28:02.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAyjI5phelgKRAAIYGXG4JEuJQ03VLyqT4j9g== 2026-02-13T02:28:02.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAyjI5phelgKRAAIYGXG4JEuJQ03VLyqT4j9g=="}' 2026-02-13T02:28:02.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4ad2bdde-fc58-4696-9ca8-10db52bfba32 -i td/osd-scrub-snaps/0/new.json 2026-02-13T02:28:02.794 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:28:02.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-snaps/0/new.json 2026-02-13T02:28:02.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=b2b95964-f5a3-4b20-bd2b-d60f1ae5f8a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.453125/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAyjI5phelgKRAAIYGXG4JEuJQ03VLyqT4j9g== --osd-uuid 4ad2bdde-fc58-4696-9ca8-10db52bfba32 2026-02-13T02:28:02.823 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:02.814+0000 7f80f6698a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:02.824 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:02.814+0000 7f80f6698a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:02.825 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:02.818+0000 7f80f6698a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:02.826 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:02.818+0000 7f80f6698a40 -1 bdev(0x562d9490ac00 td/osd-scrub-snaps/0/block) open stat got: (1) Operation not permitted 2026-02-13T02:28:02.826 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:02.818+0000 7f80f6698a40 -1 bluestore(td/osd-scrub-snaps/0) _read_fsid unparsable uuid 2026-02-13T02:28:03.536 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-snaps/0/keyring 2026-02-13T02:28:03.536 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:28:03.537 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-13T02:28:03.537 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T02:28:03.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-snaps/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:28:03.646 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-13T02:28:03.646 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T02:28:03.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=b2b95964-f5a3-4b20-bd2b-d60f1ae5f8a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.453125/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:28:03.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:28:03.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:28:03.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:28:03.680 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:03.670+0000 7f7bc9984a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:03.682 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:03.674+0000 7f7bc9984a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:03.683 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:03.674+0000 7f7bc9984a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:03.747 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-13T02:28:03.747 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:28:03.747 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:28:03.747 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:28:03.747 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:28:03.747 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:28:03.748 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:28:03.748 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:28:03.748 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:28:03.748 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:28:03.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:28:04.238 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:04.230+0000 7f7bc9984a40 -1 Falling back to public interface 2026-02-13T02:28:04.847 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:28:04.847 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:28:04.848 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:28:04.848 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:28:04.848 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:28:04.848 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:28:04.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:28:05.188 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:05.178+0000 7f7bc9984a40 -1 osd.0 0 log_to_monitors true 2026-02-13T02:28:05.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:28:05.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:28:05.944 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:28:05.944 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:28:05.944 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:28:05.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:28:06.108 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3407800482,v1:127.0.0.1:6801/3407800482] [v2:127.0.0.1:6802/3407800482,v1:127.0.0.1:6803/3407800482] exists,up 4ad2bdde-fc58-4696-9ca8-10db52bfba32 2026-02-13T02:28:06.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:28:06.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:28:06.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:28:06.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:28:06.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:744: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 1 2026-02-13T02:28:06.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-snaps 2026-02-13T02:28:06.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:28:06.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-13T02:28:06.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:28:06.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-snaps/1 2026-02-13T02:28:06.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=b2b95964-f5a3-4b20-bd2b-d60f1ae5f8a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2026-02-13T02:28:06.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:28:06.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:28:06.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:28:06.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2026-02-13T02:28:06.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2026-02-13T02:28:06.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:28:06.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:28:06.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2026-02-13T02:28:06.113 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:28:06.113 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:28:06.113 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:28:06.113 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:28:06.113 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:28:06.113 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.453125 2026-02-13T02:28:06.114 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.453125/$cluster-$name.asok' 2026-02-13T02:28:06.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.453125/$cluster-$name.asok' 2026-02-13T02:28:06.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:28:06.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:28:06.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:28:06.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2026-02-13T02:28:06.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2026-02-13T02:28:06.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:28:06.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:28:06.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:28:06.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:28:06.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:28:06.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:28:06.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-snaps/1 2026-02-13T02:28:06.115 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:28:06.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=31213213-4c5e-43d2-a538-e82c9213733e 2026-02-13T02:28:06.115 INFO:tasks.workunit.client.0.trial112.stdout:add osd1 31213213-4c5e-43d2-a538-e82c9213733e 2026-02-13T02:28:06.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 31213213-4c5e-43d2-a538-e82c9213733e' 2026-02-13T02:28:06.116 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:28:06.129 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA2jI5pB2ZKBxAANZSJoPfER/+db/Y9BBMFSQ== 2026-02-13T02:28:06.129 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA2jI5pB2ZKBxAANZSJoPfER/+db/Y9BBMFSQ=="}' 2026-02-13T02:28:06.129 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 31213213-4c5e-43d2-a538-e82c9213733e -i td/osd-scrub-snaps/1/new.json 2026-02-13T02:28:06.290 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:28:06.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-snaps/1/new.json 2026-02-13T02:28:06.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=b2b95964-f5a3-4b20-bd2b-d60f1ae5f8a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.453125/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA2jI5pB2ZKBxAANZSJoPfER/+db/Y9BBMFSQ== --osd-uuid 31213213-4c5e-43d2-a538-e82c9213733e 2026-02-13T02:28:06.323 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:06.314+0000 7fda5e2c7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:06.324 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:06.318+0000 7fda5e2c7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:06.325 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:06.318+0000 7fda5e2c7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:06.326 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:06.318+0000 7fda5e2c7a40 -1 bdev(0x55e729cb0c00 td/osd-scrub-snaps/1/block) open stat got: (1) Operation not permitted 2026-02-13T02:28:06.326 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:06.318+0000 7fda5e2c7a40 -1 bluestore(td/osd-scrub-snaps/1) _read_fsid unparsable uuid 2026-02-13T02:28:06.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-snaps/1/keyring 2026-02-13T02:28:06.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:28:06.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-13T02:28:06.956 INFO:tasks.workunit.client.0.trial112.stdout:adding osd1 key to auth repository 2026-02-13T02:28:06.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-snaps/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:28:07.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-13T02:28:07.159 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T02:28:07.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=b2b95964-f5a3-4b20-bd2b-d60f1ae5f8a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.453125/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:28:07.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:28:07.160 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:28:07.160 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:28:07.178 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:07.170+0000 7f1125f5aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:07.180 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:07.170+0000 7f1125f5aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:07.181 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:07.174+0000 7f1125f5aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:07.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-13T02:28:07.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:28:07.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:28:07.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:28:07.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:28:07.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:28:07.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:28:07.323 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:28:07.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:28:07.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:28:07.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:28:07.622 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:07.614+0000 7f1125f5aa40 -1 Falling back to public interface 2026-02-13T02:28:08.247 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:08.238+0000 7f1125f5aa40 -1 osd.1 0 log_to_monitors true 2026-02-13T02:28:08.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:28:08.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:28:08.494 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:28:08.494 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:28:08.494 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:28:08.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:28:08.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:28:09.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:28:09.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:28:09.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:28:09.662 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:28:09.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:28:09.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:28:09.827 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2324935711,v1:127.0.0.1:6811/2324935711] [v2:127.0.0.1:6812/2324935711,v1:127.0.0.1:6813/2324935711] exists,up 31213213-4c5e-43d2-a538-e82c9213733e 2026-02-13T02:28:09.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:28:09.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:28:09.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:28:09.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:748: _scrub_snaps_multi: ceph osd set noscrub 2026-02-13T02:28:10.033 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:28:10.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:749: _scrub_snaps_multi: ceph osd set nodeep-scrub 2026-02-13T02:28:10.238 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:28:10.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:752: _scrub_snaps_multi: create_pool test 1 1 2026-02-13T02:28:10.251 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2026-02-13T02:28:10.441 INFO:tasks.workunit.client.0.trial112.stderr:pool 'test' created 2026-02-13T02:28:10.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:28:11.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:753: _scrub_snaps_multi: wait_for_clean 2026-02-13T02:28:11.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:28:11.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:28:11.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:28:11.453 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:28:11.454 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:28:11.455 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:28:11.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:28:11.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:28:11.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:28:11.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:28:11.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:28:11.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:28:11.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:28:11.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:28:11.562 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:28:11.729 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:28:11.729 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:28:11.729 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:28:11.729 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:28:11.729 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:28:11.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-13T02:28:11.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-13T02:28:11.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-13T02:28:11.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:28:11.808 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:28:11.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705667 2026-02-13T02:28:11.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705667 2026-02-13T02:28:11.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705667' 2026-02-13T02:28:11.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:28:11.888 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-13T02:28:11.888 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:28:11.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:28:11.889 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-13T02:28:11.889 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:28:11.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-13T02:28:11.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-13T02:28:11.891 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836483 2026-02-13T02:28:11.891 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:28:12.064 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-13T02:28:12.064 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:28:13.065 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:28:13.066 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:28:13.230 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-13T02:28:13.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:28:13.231 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705667 2026-02-13T02:28:13.231 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:28:13.232 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:28:13.233 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705667 2026-02-13T02:28:13.233 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:28:13.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705667 2026-02-13T02:28:13.235 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705667 2026-02-13T02:28:13.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705667' 2026-02-13T02:28:13.236 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:28:13.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705667 -lt 38654705667 2026-02-13T02:28:13.414 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:28:13.414 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:28:13.415 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:28:13.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:28:13.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:28:13.609 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:28:13.609 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:28:13.609 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:28:13.609 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:28:13.610 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:28:13.610 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:28:13.768 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-13T02:28:13.769 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:28:13.769 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:28:13.769 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:28:13.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-13T02:28:13.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' -1 2026-02-13T02:28:13.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-13T02:28:13.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=0 2026-02-13T02:28:13.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-13T02:28:13.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-13T02:28:14.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-13T02:28:14.064 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:28:14.064 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:28:14.064 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:28:14.064 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:28:14.065 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:28:14.065 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:28:14.065 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:28:14.228 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-13T02:28:14.228 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:28:14.229 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:28:14.229 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:28:14.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-13T02:28:14.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-13T02:28:14.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-13T02:28:14.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-13T02:28:14.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-13T02:28:14.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-13T02:28:14.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-13T02:28:14.423 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-13T02:28:14.423 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-13T02:28:14.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-13T02:28:14.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-13T02:28:14.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-13T02:28:14.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-13T02:28:14.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-13T02:28:14.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-13T02:28:14.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:28:14.818 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:28:14.818 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:28:14.819 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:28:14.819 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:28:14.819 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:28:14.819 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:28:14.981 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:28:14.981 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:28:14.981 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:28:14.982 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:28:15.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:28:15.178 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:28:15.178 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:28:15.178 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: ceph osd dump 2026-02-13T02:28:15.179 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: grep '^pool.*['\'']test['\'']' 2026-02-13T02:28:15.179 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: awk '{ print $2 }' 2026-02-13T02:28:15.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: poolid=1 2026-02-13T02:28:15.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:756: _scrub_snaps_multi: dd if=/dev/urandom of=testdata.453125 bs=1032 count=1 2026-02-13T02:28:15.341 INFO:tasks.workunit.client.0.trial112.stderr:1+0 records in 2026-02-13T02:28:15.341 INFO:tasks.workunit.client.0.trial112.stderr:1+0 records out 2026-02-13T02:28:15.341 INFO:tasks.workunit.client.0.trial112.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 3.738e-05 s, 27.6 MB/s 2026-02-13T02:28:15.341 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: seq 1 16 2026-02-13T02:28:15.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-13T02:28:15.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj1 testdata.453125 2026-02-13T02:28:15.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-13T02:28:15.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj2 testdata.453125 2026-02-13T02:28:15.393 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-13T02:28:15.394 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj3 testdata.453125 2026-02-13T02:28:15.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-13T02:28:15.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj4 testdata.453125 2026-02-13T02:28:15.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-13T02:28:15.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj5 testdata.453125 2026-02-13T02:28:15.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-13T02:28:15.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj6 testdata.453125 2026-02-13T02:28:15.511 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-13T02:28:15.511 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj7 testdata.453125 2026-02-13T02:28:15.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-13T02:28:15.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj8 testdata.453125 2026-02-13T02:28:15.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-13T02:28:15.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj9 testdata.453125 2026-02-13T02:28:15.597 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-13T02:28:15.597 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj10 testdata.453125 2026-02-13T02:28:15.626 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-13T02:28:15.626 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj11 testdata.453125 2026-02-13T02:28:15.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-13T02:28:15.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj12 testdata.453125 2026-02-13T02:28:15.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-13T02:28:15.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj13 testdata.453125 2026-02-13T02:28:15.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-13T02:28:15.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj14 testdata.453125 2026-02-13T02:28:15.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-13T02:28:15.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj15 testdata.453125 2026-02-13T02:28:15.771 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-13T02:28:15.771 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj16 testdata.453125 2026-02-13T02:28:15.799 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:762: _scrub_snaps_multi: get_primary test obj1 2026-02-13T02:28:15.799 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2026-02-13T02:28:15.800 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-13T02:28:15.800 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2026-02-13T02:28:15.800 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T02:28:15.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:762: _scrub_snaps_multi: local primary=1 2026-02-13T02:28:15.966 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:763: _scrub_snaps_multi: get_not_primary test obj1 2026-02-13T02:28:15.966 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=test 2026-02-13T02:28:15.966 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=obj1 2026-02-13T02:28:15.967 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary test obj1 2026-02-13T02:28:15.967 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2026-02-13T02:28:15.967 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-13T02:28:15.967 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2026-02-13T02:28:15.967 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T02:28:16.133 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-13T02:28:16.134 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map test obj1 2026-02-13T02:28:16.134 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-13T02:28:16.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:763: _scrub_snaps_multi: local replica=0 2026-02-13T02:28:16.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:765: _scrub_snaps_multi: eval create_scenario td/osd-scrub-snaps test testdata.453125 '$replica' 2026-02-13T02:28:16.303 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:765: _scrub_snaps_multi: create_scenario td/osd-scrub-snaps test testdata.453125 0 2026-02-13T02:28:16.303 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:45: create_scenario: local dir=td/osd-scrub-snaps 2026-02-13T02:28:16.303 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:46: create_scenario: local poolname=test 2026-02-13T02:28:16.303 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:47: create_scenario: local TESTDATA=testdata.453125 2026-02-13T02:28:16.303 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:48: create_scenario: local osd=0 2026-02-13T02:28:16.303 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:50: create_scenario: SNAP=1 2026-02-13T02:28:16.303 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:51: create_scenario: rados -p test mksnap snap1 2026-02-13T02:28:16.374 INFO:tasks.workunit.client.0.trial112.stdout:created pool test snap snap1 2026-02-13T02:28:16.375 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:52: create_scenario: dd if=/dev/urandom of=testdata.453125 bs=256 count=1 2026-02-13T02:28:16.377 INFO:tasks.workunit.client.0.trial112.stderr:1+0 records in 2026-02-13T02:28:16.377 INFO:tasks.workunit.client.0.trial112.stderr:1+0 records out 2026-02-13T02:28:16.377 INFO:tasks.workunit.client.0.trial112.stderr:256 bytes copied, 0.000100347 s, 2.6 MB/s 2026-02-13T02:28:16.377 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:53: create_scenario: rados -p test put obj1 testdata.453125 2026-02-13T02:28:16.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:54: create_scenario: rados -p test put obj5 testdata.453125 2026-02-13T02:28:16.435 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:55: create_scenario: rados -p test put obj3 testdata.453125 2026-02-13T02:28:16.462 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: seq 6 14 2026-02-13T02:28:16.463 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-13T02:28:16.463 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj6 testdata.453125 2026-02-13T02:28:16.485 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-13T02:28:16.486 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj7 testdata.453125 2026-02-13T02:28:16.515 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-13T02:28:16.515 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj8 testdata.453125 2026-02-13T02:28:16.543 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-13T02:28:16.543 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj9 testdata.453125 2026-02-13T02:28:16.572 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-13T02:28:16.572 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj10 testdata.453125 2026-02-13T02:28:16.594 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-13T02:28:16.594 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj11 testdata.453125 2026-02-13T02:28:16.623 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-13T02:28:16.623 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj12 testdata.453125 2026-02-13T02:28:16.651 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-13T02:28:16.651 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj13 testdata.453125 2026-02-13T02:28:16.681 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-13T02:28:16.681 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj14 testdata.453125 2026-02-13T02:28:16.707 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:60: create_scenario: SNAP=2 2026-02-13T02:28:16.707 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:61: create_scenario: rados -p test mksnap snap2 2026-02-13T02:28:16.786 INFO:tasks.workunit.client.0.trial112.stdout:created pool test snap snap2 2026-02-13T02:28:16.788 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:62: create_scenario: dd if=/dev/urandom of=testdata.453125 bs=256 count=2 2026-02-13T02:28:16.789 INFO:tasks.workunit.client.0.trial112.stderr:2+0 records in 2026-02-13T02:28:16.790 INFO:tasks.workunit.client.0.trial112.stderr:2+0 records out 2026-02-13T02:28:16.790 INFO:tasks.workunit.client.0.trial112.stderr:512 bytes copied, 9.0569e-05 s, 5.7 MB/s 2026-02-13T02:28:16.790 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:63: create_scenario: rados -p test put obj5 testdata.453125 2026-02-13T02:28:16.821 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:65: create_scenario: SNAP=3 2026-02-13T02:28:16.821 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:66: create_scenario: rados -p test mksnap snap3 2026-02-13T02:28:16.887 INFO:tasks.workunit.client.0.trial112.stdout:created pool test snap snap3 2026-02-13T02:28:16.890 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:67: create_scenario: dd if=/dev/urandom of=testdata.453125 bs=256 count=3 2026-02-13T02:28:16.891 INFO:tasks.workunit.client.0.trial112.stderr:3+0 records in 2026-02-13T02:28:16.891 INFO:tasks.workunit.client.0.trial112.stderr:3+0 records out 2026-02-13T02:28:16.891 INFO:tasks.workunit.client.0.trial112.stderr:768 bytes copied, 9.3905e-05 s, 8.2 MB/s 2026-02-13T02:28:16.892 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:68: create_scenario: rados -p test put obj3 testdata.453125 2026-02-13T02:28:16.913 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:70: create_scenario: SNAP=4 2026-02-13T02:28:16.913 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:71: create_scenario: rados -p test mksnap snap4 2026-02-13T02:28:16.990 INFO:tasks.workunit.client.0.trial112.stdout:created pool test snap snap4 2026-02-13T02:28:16.992 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:72: create_scenario: dd if=/dev/urandom of=testdata.453125 bs=256 count=4 2026-02-13T02:28:16.993 INFO:tasks.workunit.client.0.trial112.stderr:4+0 records in 2026-02-13T02:28:16.993 INFO:tasks.workunit.client.0.trial112.stderr:4+0 records out 2026-02-13T02:28:16.994 INFO:tasks.workunit.client.0.trial112.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000101459 s, 10.1 MB/s 2026-02-13T02:28:16.994 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:73: create_scenario: rados -p test put obj5 testdata.453125 2026-02-13T02:28:17.022 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:74: create_scenario: rados -p test put obj2 testdata.453125 2026-02-13T02:28:17.050 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:76: create_scenario: SNAP=5 2026-02-13T02:28:17.050 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:77: create_scenario: rados -p test mksnap snap5 2026-02-13T02:28:17.090 INFO:tasks.workunit.client.0.trial112.stdout:created pool test snap snap5 2026-02-13T02:28:17.093 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:78: create_scenario: SNAP=6 2026-02-13T02:28:17.093 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:79: create_scenario: rados -p test mksnap snap6 2026-02-13T02:28:17.193 INFO:tasks.workunit.client.0.trial112.stdout:created pool test snap snap6 2026-02-13T02:28:17.195 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:80: create_scenario: dd if=/dev/urandom of=testdata.453125 bs=256 count=6 2026-02-13T02:28:17.196 INFO:tasks.workunit.client.0.trial112.stderr:6+0 records in 2026-02-13T02:28:17.196 INFO:tasks.workunit.client.0.trial112.stderr:6+0 records out 2026-02-13T02:28:17.196 INFO:tasks.workunit.client.0.trial112.stderr:1536 bytes (1.5 kB, 1.5 KiB) copied, 0.000104024 s, 14.8 MB/s 2026-02-13T02:28:17.197 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:81: create_scenario: rados -p test put obj5 testdata.453125 2026-02-13T02:28:17.226 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:83: create_scenario: SNAP=7 2026-02-13T02:28:17.226 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:84: create_scenario: rados -p test mksnap snap7 2026-02-13T02:28:17.295 INFO:tasks.workunit.client.0.trial112.stdout:created pool test snap snap7 2026-02-13T02:28:17.297 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:86: create_scenario: rados -p test rm obj4 2026-02-13T02:28:17.320 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:87: create_scenario: rados -p test rm obj16 2026-02-13T02:28:17.346 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:88: create_scenario: rados -p test rm obj2 2026-02-13T02:28:17.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:90: create_scenario: kill_daemons td/osd-scrub-snaps TERM osd 2026-02-13T02:28:17.374 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:28:17.374 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:28:17.375 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:28:17.375 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:28:17.375 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:28:17.683 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:28:17.684 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj1 2026-02-13T02:28:17.884 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: JSON='["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:28:17.884 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:95: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' --force remove 2026-02-13T02:28:18.083 INFO:tasks.workunit.client.0.trial112.stdout:WARNING: only removing head with clones present 2026-02-13T02:28:18.084 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:ff7b1f36:::obj1:head# 2026-02-13T02:28:18.177 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2026-02-13T02:28:18.177 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: grep '"snapid":2' 2026-02-13T02:28:18.344 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:28:18.344 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:98: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2026-02-13T02:28:18.532 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:c52c9666:::obj5:2# 2026-02-13T02:28:18.621 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2026-02-13T02:28:18.621 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: grep '"snapid":1' 2026-02-13T02:28:18.816 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:28:18.816 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:101: create_scenario: OBJ5SAVE='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:28:18.816 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:103: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2026-02-13T02:28:19.043 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:104: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2026-02-13T02:28:19.045 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:28:19.045 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2026-02-13T02:28:19.045 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:28:19.045 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-13T02:28:19.045 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2026-02-13T02:28:19.045 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2026-02-13T02:28:19.045 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2026-02-13T02:28:19.045 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2026-02-13T02:28:19.046 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2026-02-13T02:28:19.046 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2026-02-13T02:28:19.046 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:28:19.046 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2026-02-13T02:28:19.046 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2026-02-13T02:28:19.046 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2026-02-13T02:28:19.046 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2026-02-13T02:28:19.046 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:28:19.046 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2026-02-13T02:28:19.046 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:28:19.046 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:28:19.047 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:28:19.047 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2026-02-13T02:28:19.047 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:28:19.047 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2026-02-13T02:28:19.047 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:28:19.047 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:28:19.047 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:28:19.047 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2026-02-13T02:28:19.047 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:28:19.047 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:28:19.047 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:28:19.047 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2026-02-13T02:28:19.048 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2026-02-13T02:28:19.048 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:28:19.048 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:28:19.048 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2026-02-13T02:28:19.048 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2026-02-13T02:28:19.048 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:28:19.048 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:28:19.048 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2026-02-13T02:28:19.048 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:105: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2026-02-13T02:28:19.049 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-13T02:28:19.049 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:106: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype nosnapmap '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2026-02-13T02:28:19.212 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:c52c9666:::obj5:1# 2026-02-13T02:28:19.308 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:108: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2026-02-13T02:28:19.535 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:109: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2026-02-13T02:28:19.537 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:28:19.537 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2026-02-13T02:28:19.537 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:28:19.537 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-13T02:28:19.537 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2026-02-13T02:28:19.537 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2026-02-13T02:28:19.537 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2026-02-13T02:28:19.537 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2026-02-13T02:28:19.537 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2026-02-13T02:28:19.537 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2026-02-13T02:28:19.538 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:28:19.538 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2026-02-13T02:28:19.538 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2026-02-13T02:28:19.538 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2026-02-13T02:28:19.538 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2026-02-13T02:28:19.538 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:28:19.538 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2026-02-13T02:28:19.538 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:28:19.538 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:28:19.538 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:28:19.538 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2026-02-13T02:28:19.538 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:28:19.539 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2026-02-13T02:28:19.539 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:28:19.539 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:28:19.539 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:28:19.539 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2026-02-13T02:28:19.539 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:28:19.539 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:28:19.539 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:28:19.539 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2026-02-13T02:28:19.539 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2026-02-13T02:28:19.539 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:28:19.539 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:28:19.539 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2026-02-13T02:28:19.540 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2026-02-13T02:28:19.540 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2026-02-13T02:28:19.540 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:28:19.540 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2026-02-13T02:28:19.540 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:110: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2026-02-13T02:28:19.541 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-13T02:28:19.541 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:111: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2026-02-13T02:28:19.541 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2026-02-13T02:28:19.541 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: grep '"snapid":4' 2026-02-13T02:28:19.704 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:28:19.705 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:114: create_scenario: dd if=/dev/urandom of=testdata.453125 bs=256 count=18 2026-02-13T02:28:19.706 INFO:tasks.workunit.client.0.trial112.stderr:18+0 records in 2026-02-13T02:28:19.706 INFO:tasks.workunit.client.0.trial112.stderr:18+0 records out 2026-02-13T02:28:19.706 INFO:tasks.workunit.client.0.trial112.stderr:4608 bytes (4.6 kB, 4.5 KiB) copied, 0.000299701 s, 15.4 MB/s 2026-02-13T02:28:19.706 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:115: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.453125 2026-02-13T02:28:19.977 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj3 2026-02-13T02:28:20.132 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: JSON='["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:28:20.132 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:118: create_scenario: dd if=/dev/urandom of=testdata.453125 bs=256 count=15 2026-02-13T02:28:20.133 INFO:tasks.workunit.client.0.trial112.stderr:15+0 records in 2026-02-13T02:28:20.133 INFO:tasks.workunit.client.0.trial112.stderr:15+0 records out 2026-02-13T02:28:20.133 INFO:tasks.workunit.client.0.trial112.stderr:3840 bytes (3.8 kB, 3.8 KiB) copied, 8.532e-05 s, 45.0 MB/s 2026-02-13T02:28:20.133 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:119: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.453125 2026-02-13T02:28:20.393 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj4 2026-02-13T02:28:20.393 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: grep '"snapid":7' 2026-02-13T02:28:20.576 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: JSON='["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:28:20.576 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:122: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' remove 2026-02-13T02:28:20.772 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:0ee9ae15:::obj4:7# 2026-02-13T02:28:20.876 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:125: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2026-02-13T02:28:21.107 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:126: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2026-02-13T02:28:21.109 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2026-02-13T02:28:21.109 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:28:21.109 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-13T02:28:21.109 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2026-02-13T02:28:21.109 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2026-02-13T02:28:21.109 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2026-02-13T02:28:21.109 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2026-02-13T02:28:21.109 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2026-02-13T02:28:21.109 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2026-02-13T02:28:21.110 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:28:21.110 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2026-02-13T02:28:21.110 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2026-02-13T02:28:21.110 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2026-02-13T02:28:21.110 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2026-02-13T02:28:21.110 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2026-02-13T02:28:21.110 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:28:21.110 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:28:21.110 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2026-02-13T02:28:21.110 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:28:21.110 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2026-02-13T02:28:21.111 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:28:21.111 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:28:21.111 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2026-02-13T02:28:21.111 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:28:21.111 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:28:21.111 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2026-02-13T02:28:21.111 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2026-02-13T02:28:21.111 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:28:21.111 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2026-02-13T02:28:21.111 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2026-02-13T02:28:21.111 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:28:21.111 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2026-02-13T02:28:21.112 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:127: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2026-02-13T02:28:21.112 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:28:21.112 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:28:21.112 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:28:21.112 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:28:21.113 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:28:21.113 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:28:21.113 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2026-02-13T02:28:21.113 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj16 2026-02-13T02:28:21.113 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: grep '"snapid":7' 2026-02-13T02:28:21.284 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: JSON='["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:28:21.284 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:129: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype snapmap '["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' remove 2026-02-13T02:28:21.479 INFO:tasks.workunit.client.0.trial112.stdout:remove #1:461f8b5e:::obj16:7# 2026-02-13T02:28:21.570 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:131: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2026-02-13T02:28:21.783 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:132: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2026-02-13T02:28:21.785 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2026-02-13T02:28:21.785 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-13T02:28:21.785 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2026-02-13T02:28:21.785 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2026-02-13T02:28:21.785 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2026-02-13T02:28:21.785 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2026-02-13T02:28:21.785 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2026-02-13T02:28:21.785 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2026-02-13T02:28:21.785 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:28:21.786 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2026-02-13T02:28:21.786 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2026-02-13T02:28:21.786 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2026-02-13T02:28:21.786 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2026-02-13T02:28:21.786 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2026-02-13T02:28:21.786 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:28:21.786 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2026-02-13T02:28:21.786 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2026-02-13T02:28:21.786 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:28:21.786 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2026-02-13T02:28:21.786 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2026-02-13T02:28:21.786 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2026-02-13T02:28:21.786 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2026-02-13T02:28:21.787 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2026-02-13T02:28:21.787 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2026-02-13T02:28:21.787 INFO:tasks.workunit.client.0.trial112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2026-02-13T02:28:21.787 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:133: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2026-02-13T02:28:21.787 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:134: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2026-02-13T02:28:21.788 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj2 2026-02-13T02:28:21.972 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: JSON='["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:28:21.972 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:137: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' rm-attr snapset 2026-02-13T02:28:22.253 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: echo '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:28:22.254 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: sed 's/snapid":1/snapid":7/' 2026-02-13T02:28:22.255 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:28:22.255 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:141: create_scenario: dd if=/dev/urandom of=testdata.453125 bs=256 count=7 2026-02-13T02:28:22.256 INFO:tasks.workunit.client.0.trial112.stderr:7+0 records in 2026-02-13T02:28:22.256 INFO:tasks.workunit.client.0.trial112.stderr:7+0 records out 2026-02-13T02:28:22.256 INFO:tasks.workunit.client.0.trial112.stderr:1792 bytes (1.8 kB, 1.8 KiB) copied, 8.5551e-05 s, 20.9 MB/s 2026-02-13T02:28:22.257 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:142: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.453125 2026-02-13T02:28:22.537 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj6 2026-02-13T02:28:22.619 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-13T02:28:22.720 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: JSON='["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:28:22.720 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:145: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset 2026-02-13T02:28:22.969 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj7 2026-02-13T02:28:23.039 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-13T02:28:23.140 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: JSON='["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:28:23.140 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:147: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset corrupt 2026-02-13T02:28:23.429 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj8 2026-02-13T02:28:23.503 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-13T02:28:23.587 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: JSON='["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:28:23.588 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:149: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset seq 2026-02-13T02:28:23.829 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj9 2026-02-13T02:28:23.915 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-13T02:28:24.020 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: JSON='["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:28:24.020 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:151: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_size 2026-02-13T02:28:24.281 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj10 2026-02-13T02:28:24.359 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-13T02:28:24.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: JSON='["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:28:24.456 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:153: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_overlap 2026-02-13T02:28:24.742 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj11 2026-02-13T02:28:24.816 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-13T02:28:24.908 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: JSON='["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:28:24.908 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:155: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clones 2026-02-13T02:28:25.221 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj12 2026-02-13T02:28:25.303 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-13T02:28:25.415 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: JSON='["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:28:25.416 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:157: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset head 2026-02-13T02:28:25.717 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj13 2026-02-13T02:28:25.783 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-13T02:28:25.896 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: JSON='["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:28:25.896 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:159: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset snaps 2026-02-13T02:28:26.178 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj14 2026-02-13T02:28:26.259 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-13T02:28:26.335 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: JSON='["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:28:26.336 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:161: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset size 2026-02-13T02:28:26.580 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:163: create_scenario: echo garbage 2026-02-13T02:28:26.581 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj15 2026-02-13T02:28:26.662 INFO:tasks.workunit.client.0.trial112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-13T02:28:26.752 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: JSON='["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:28:26.752 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:165: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' set-attr snapset td/osd-scrub-snaps/bad 2026-02-13T02:28:27.016 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:166: create_scenario: rm -f td/osd-scrub-snaps/bad 2026-02-13T02:28:27.017 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:167: create_scenario: return 0 2026-02-13T02:28:27.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:767: _scrub_snaps_multi: rm -f testdata.453125 2026-02-13T02:28:27.019 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: expr 2 - 1 2026-02-13T02:28:27.020 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: seq 0 1 2026-02-13T02:28:27.024 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:28:27.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:771: _scrub_snaps_multi: activate_osd td/osd-scrub-snaps 0 2026-02-13T02:28:27.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-snaps 2026-02-13T02:28:27.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:28:27.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-13T02:28:27.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:28:27.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-snaps/0 2026-02-13T02:28:27.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=b2b95964-f5a3-4b20-bd2b-d60f1ae5f8a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2026-02-13T02:28:27.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:28:27.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:28:27.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:28:27.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2026-02-13T02:28:27.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2026-02-13T02:28:27.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:28:27.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:28:27.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2026-02-13T02:28:27.026 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:28:27.026 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:28:27.026 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:28:27.026 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:28:27.026 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:28:27.027 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.453125 2026-02-13T02:28:27.027 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.453125/$cluster-$name.asok' 2026-02-13T02:28:27.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.453125/$cluster-$name.asok' 2026-02-13T02:28:27.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:28:27.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2026-02-13T02:28:27.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2026-02-13T02:28:27.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:28:27.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:28:27.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:28:27.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:28:27.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:28:27.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:28:27.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-snaps/0 2026-02-13T02:28:27.028 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T02:28:27.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-13T02:28:27.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=b2b95964-f5a3-4b20-bd2b-d60f1ae5f8a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.453125/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:28:27.028 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-snaps/0/whoami 2026-02-13T02:28:27.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-13T02:28:27.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:28:27.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:28:27.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:28:27.046 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:27.038+0000 7fdfddbe0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:27.048 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:27.038+0000 7fdfddbe0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:27.049 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:27.042+0000 7fdfddbe0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:27.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-13T02:28:27.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:28:27.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:28:27.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:28:27.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:28:27.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:28:27.190 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:28:27.190 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:28:27.190 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:28:27.190 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:28:27.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:28:27.466 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:27.458+0000 7fdfddbe0a40 -1 Falling back to public interface 2026-02-13T02:28:27.999 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:27.990+0000 7fdfddbe0a40 -1 osd.0 21 log_to_monitors true 2026-02-13T02:28:28.357 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:28:28.357 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:28:28.357 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:28:28.357 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:28:28.358 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:28:28.358 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:28:28.522 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 24 up_thru 24 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6800/3362533249,v1:127.0.0.1:6801/3362533249] [v2:127.0.0.1:6802/3362533249,v1:127.0.0.1:6803/3362533249] exists,up 4ad2bdde-fc58-4696-9ca8-10db52bfba32 2026-02-13T02:28:28.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:28:28.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:28:28.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:28:28.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:28:28.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:771: _scrub_snaps_multi: activate_osd td/osd-scrub-snaps 1 2026-02-13T02:28:28.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-snaps 2026-02-13T02:28:28.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:28:28.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:28:28.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:28:28.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-snaps/1 2026-02-13T02:28:28.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=b2b95964-f5a3-4b20-bd2b-d60f1ae5f8a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2026-02-13T02:28:28.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:28:28.527 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:28:28.527 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:28:28.527 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2026-02-13T02:28:28.527 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2026-02-13T02:28:28.527 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:28:28.527 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:28:28.527 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2026-02-13T02:28:28.527 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:28:28.527 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:28:28.527 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:28:28.527 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:28:28.527 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:28:28.528 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.453125 2026-02-13T02:28:28.528 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.453125/$cluster-$name.asok' 2026-02-13T02:28:28.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.453125/$cluster-$name.asok' 2026-02-13T02:28:28.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:28:28.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2026-02-13T02:28:28.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2026-02-13T02:28:28.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:28:28.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:28:28.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:28:28.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:28:28.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:28:28.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:28:28.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-snaps/1 2026-02-13T02:28:28.529 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T02:28:28.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:28:28.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=b2b95964-f5a3-4b20-bd2b-d60f1ae5f8a0 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.453125/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:28:28.530 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-snaps/1/whoami 2026-02-13T02:28:28.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:28:28.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:28:28.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:28:28.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:28:28.553 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:28.546+0000 7fbcf0855a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:28.555 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:28.546+0000 7fbcf0855a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:28.556 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:28.546+0000 7fbcf0855a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:28.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:28:28.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:28:28.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:28:28.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:28:28.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:28:28.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:28:28.690 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:28:28.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:28:28.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:28:28.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:28:28.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:28:28.938 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:28.930+0000 7fbcf0855a40 -1 Falling back to public interface 2026-02-13T02:28:29.546 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:29.538+0000 7fbcf0855a40 -1 osd.1 21 log_to_monitors true 2026-02-13T02:28:29.859 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:28:29.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:28:29.860 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:28:29.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:28:29.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:28:29.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:28:30.023 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 28 up_thru 28 down_at 22 last_clean_interval [9,21) [v2:127.0.0.1:6810/4281917019,v1:127.0.0.1:6811/4281917019] [v2:127.0.0.1:6812/4281917019,v1:127.0.0.1:6813/4281917019] exists,up 31213213-4c5e-43d2-a538-e82c9213733e 2026-02-13T02:28:30.024 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:28:30.024 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:28:30.024 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:28:30.024 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:774: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_max 3 2026-02-13T02:28:30.094 INFO:tasks.workunit.client.0.trial112.stdout:osd.0: { 2026-02-13T02:28:30.094 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-13T02:28:30.095 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:28:30.102 INFO:tasks.workunit.client.0.trial112.stdout:osd.1: { 2026-02-13T02:28:30.102 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-13T02:28:30.102 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:28:30.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:775: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_min 3 2026-02-13T02:28:30.183 INFO:tasks.workunit.client.0.trial112.stdout:osd.0: { 2026-02-13T02:28:30.183 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_shallow_scrub_chunk_min = '' " 2026-02-13T02:28:30.183 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:28:30.190 INFO:tasks.workunit.client.0.trial112.stdout:osd.1: { 2026-02-13T02:28:30.190 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_shallow_scrub_chunk_min = '' " 2026-02-13T02:28:30.190 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:28:30.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:776: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_scrub_chunk_min 3 2026-02-13T02:28:30.269 INFO:tasks.workunit.client.0.trial112.stdout:osd.0: { 2026-02-13T02:28:30.269 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_chunk_min = '' " 2026-02-13T02:28:30.269 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:28:30.277 INFO:tasks.workunit.client.0.trial112.stdout:osd.1: { 2026-02-13T02:28:30.277 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_chunk_min = '' " 2026-02-13T02:28:30.277 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:28:30.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:777: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_seconds 1 2026-02-13T02:28:30.364 INFO:tasks.workunit.client.0.trial112.stdout:osd.0: { 2026-02-13T02:28:30.364 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_pg_stat_report_interval_max_seconds = '' " 2026-02-13T02:28:30.365 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:28:30.372 INFO:tasks.workunit.client.0.trial112.stdout:osd.1: { 2026-02-13T02:28:30.372 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_pg_stat_report_interval_max_seconds = '' " 2026-02-13T02:28:30.372 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:28:30.382 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:778: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_epochs 1 2026-02-13T02:28:30.452 INFO:tasks.workunit.client.0.trial112.stdout:osd.0: { 2026-02-13T02:28:30.453 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_pg_stat_report_interval_max_epochs = '' " 2026-02-13T02:28:30.453 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:28:30.460 INFO:tasks.workunit.client.0.trial112.stdout:osd.1: { 2026-02-13T02:28:30.460 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_pg_stat_report_interval_max_epochs = '' " 2026-02-13T02:28:30.460 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:28:30.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:779: _scrub_snaps_multi: wait_for_clean 2026-02-13T02:28:30.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:28:30.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:28:30.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:28:30.469 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:28:30.469 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:28:30.469 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:28:30.469 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:28:30.470 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:28:30.470 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:28:30.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:28:30.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:28:30.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:28:30.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:28:30.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:28:30.569 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:28:30.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:28:30.731 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:28:30.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:28:30.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:28:30.732 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:28:30.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215106 2026-02-13T02:28:30.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215106 2026-02-13T02:28:30.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215106' 2026-02-13T02:28:30.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:28:30.812 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:28:30.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=120259084290 2026-02-13T02:28:30.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 120259084290 2026-02-13T02:28:30.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215106 1-120259084290' 2026-02-13T02:28:30.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:28:30.890 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-103079215106 2026-02-13T02:28:30.890 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:28:30.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:28:30.892 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-103079215106 2026-02-13T02:28:30.892 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:28:30.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215106 2026-02-13T02:28:30.894 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 103079215106 2026-02-13T02:28:30.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2026-02-13T02:28:30.894 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:28:31.066 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215106 -lt 103079215106 2026-02-13T02:28:31.066 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:28:31.067 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-120259084290 2026-02-13T02:28:31.067 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:28:31.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:28:31.068 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-120259084290 2026-02-13T02:28:31.068 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:28:31.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=120259084290 2026-02-13T02:28:31.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 120259084290' 2026-02-13T02:28:31.070 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 120259084290 2026-02-13T02:28:31.070 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:28:31.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 120259084290 -lt 120259084290 2026-02-13T02:28:31.247 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:28:31.248 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:28:31.248 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:28:31.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:28:31.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:28:31.454 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:28:31.454 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:28:31.454 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:28:31.454 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:28:31.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:28:31.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:28:31.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:28:31.623 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:28:31.624 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:28:31.624 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:28:31.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:28:31.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:28:31.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:28:31.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:781: _scrub_snaps_multi: local pgid=1.0 2026-02-13T02:28:31.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:782: _scrub_snaps_multi: pg_scrub 1.0 2026-02-13T02:28:31.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=1.0 2026-02-13T02:28:31.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 1.0 2026-02-13T02:28:31.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=1.0 2026-02-13T02:28:31.824 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-13T02:28:31.824 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:28:31.824 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:28:31.825 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:28:31.825 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:28:31.825 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:28:32.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-13T02:28:32.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-13T02:28:32.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-13T02:28:32.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-13T02:28:32.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:28:32.090 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:28:32.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:28:32.259 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:28:32.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:28:32.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:28:32.259 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:28:32.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215107 2026-02-13T02:28:32.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215107 2026-02-13T02:28:32.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215107' 2026-02-13T02:28:32.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:28:32.339 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:28:32.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=120259084291 2026-02-13T02:28:32.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 120259084291 2026-02-13T02:28:32.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215107 1-120259084291' 2026-02-13T02:28:32.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:28:32.418 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-103079215107 2026-02-13T02:28:32.418 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:28:32.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:28:32.420 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-103079215107 2026-02-13T02:28:32.420 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:28:32.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215107 2026-02-13T02:28:32.421 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 103079215107 2026-02-13T02:28:32.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 103079215107' 2026-02-13T02:28:32.422 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:28:32.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215106 -lt 103079215107 2026-02-13T02:28:32.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:28:33.596 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:28:33.597 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:28:33.766 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215107 -lt 103079215107 2026-02-13T02:28:33.766 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:28:33.767 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-120259084291 2026-02-13T02:28:33.767 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:28:33.768 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:28:33.769 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-120259084291 2026-02-13T02:28:33.769 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:28:33.770 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=120259084291 2026-02-13T02:28:33.770 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 120259084291' 2026-02-13T02:28:33.771 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 120259084291 2026-02-13T02:28:33.771 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:28:33.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 120259084291 -lt 120259084291 2026-02-13T02:28:33.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-13T02:28:33.942 INFO:tasks.workunit.client.0.trial112.stdout:#---------- 1.0 loop 0 2026-02-13T02:28:33.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 0' 2026-02-13T02:28:33.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-13T02:28:33.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-13T02:28:33.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-13T02:28:33.944 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-13T02:28:33.944 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-13T02:28:34.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-13T02:28:34.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-13T02:28:34.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-13T02:28:34.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-13T02:28:34.020 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: get_last_scrub_stamp 1.0 2026-02-13T02:28:34.021 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:28:34.021 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:28:34.021 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:28:34.021 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:28:34.182 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: local last_scrub=2026-02-13T02:28:10.433137+0000 2026-02-13T02:28:34.182 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_scrub: ceph pg scrub 1.0 2026-02-13T02:28:34.342 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 1.0 on osd.1 to scrub 2026-02-13T02:28:34.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_scrub: wait_for_scrub 1.0 2026-02-13T02:28:10.433137+0000 2026-02-13T02:28:34.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-13T02:28:34.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:28:10.433137+0000 2026-02-13T02:28:34.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:28:34.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:28:34.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:28:34.353 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:28:34.353 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:28:34.353 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:28:34.353 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:28:34.354 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:28:34.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:28:10.433137+0000 '>' 2026-02-13T02:28:10.433137+0000 2026-02-13T02:28:34.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:28:35.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:28:35.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:28:35.522 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:28:35.522 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:28:35.522 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:28:35.523 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:28:35.523 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:28:35.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:28:10.433137+0000 '>' 2026-02-13T02:28:10.433137+0000 2026-02-13T02:28:35.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:28:36.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:28:36.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:28:36.687 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:28:36.687 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:28:36.687 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:28:36.688 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:28:36.688 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:28:36.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:28:10.433137+0000 '>' 2026-02-13T02:28:10.433137+0000 2026-02-13T02:28:36.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:28:37.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:28:37.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:28:37.851 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:28:37.851 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:28:37.851 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:28:37.851 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:28:37.851 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:28:38.015 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:28:10.433137+0000 '>' 2026-02-13T02:28:10.433137+0000 2026-02-13T02:28:38.015 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:28:39.016 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:28:39.016 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:28:39.017 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:28:39.017 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:28:39.017 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:28:39.017 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:28:39.017 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:28:39.176 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:28:10.433137+0000 '>' 2026-02-13T02:28:10.433137+0000 2026-02-13T02:28:39.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:28:40.178 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:28:40.178 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:28:40.178 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:28:40.178 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:28:40.179 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:28:40.179 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:28:40.179 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:28:40.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:28:10.433137+0000 '>' 2026-02-13T02:28:10.433137+0000 2026-02-13T02:28:40.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:28:41.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:28:41.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-13T02:28:41.347 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:28:41.347 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:28:41.347 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:28:41.347 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:28:41.347 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:28:41.512 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:28:34.625388+0000 '>' 2026-02-13T02:28:10.433137+0000 2026-02-13T02:28:41.513 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:28:41.513 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: grep '_scan_snaps start' td/osd-scrub-snaps/osd.1.log 2026-02-13T02:28:41.513 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: wc -l 2026-02-13T02:28:41.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: test 16 -gt 3 2026-02-13T02:28:41.517 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: grep '_scan_snaps start' td/osd-scrub-snaps/osd.0.log 2026-02-13T02:28:41.517 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: wc -l 2026-02-13T02:28:41.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: test 16 -gt 3 2026-02-13T02:28:41.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:789: _scrub_snaps_multi: rados list-inconsistent-pg test 2026-02-13T02:28:41.546 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:791: _scrub_snaps_multi: jq '. | length' td/osd-scrub-snaps/json 2026-02-13T02:28:41.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:791: _scrub_snaps_multi: test 1 = 1 2026-02-13T02:28:41.559 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:793: _scrub_snaps_multi: jq -r '.[0]' td/osd-scrub-snaps/json 2026-02-13T02:28:41.568 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:793: _scrub_snaps_multi: test 1.0 = 1.0 2026-02-13T02:28:41.568 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:795: _scrub_snaps_multi: rados list-inconsistent-obj 1.0 --format=json-pretty 2026-02-13T02:28:41.588 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:28:41.588 INFO:tasks.workunit.client.0.trial112.stdout: "epoch": 28, 2026-02-13T02:28:41.588 INFO:tasks.workunit.client.0.trial112.stdout: "inconsistents": [ 2026-02-13T02:28:41.588 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.588 INFO:tasks.workunit.client.0.trial112.stdout: "object": { 2026-02-13T02:28:41.588 INFO:tasks.workunit.client.0.trial112.stdout: "name": "obj1", 2026-02-13T02:28:41.588 INFO:tasks.workunit.client.0.trial112.stdout: "nspace": "", 2026-02-13T02:28:41.588 INFO:tasks.workunit.client.0.trial112.stdout: "locator": "", 2026-02-13T02:28:41.588 INFO:tasks.workunit.client.0.trial112.stdout: "snap": "head", 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: "version": 18 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: "union_shard_errors": [ 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: "missing" 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: "selected_object_info": { 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: "oid": { 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: "oid": "obj1", 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: "key": "", 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: "snapid": -2, 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: "hash": 1828249343, 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: "max": 0, 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: "pool": 1, 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: "namespace": "" 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: "version": "15'18", 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: "prior_version": "14'1", 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: "last_reqid": "client.4232.0:1", 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: "user_version": 18, 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: "mtime": "2026-02-13T02:28:16.397771+0000", 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: "local_mtime": "2026-02-13T02:28:16.399207+0000", 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: "lost": 0, 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: "flags": [ 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: "dirty", 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest" 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_seq": 0, 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_size": 0, 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest": "0xaa2f2450", 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: "omap_digest": "0xffffffff", 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: "expected_object_size": 0, 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: "expected_write_size": 0, 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: "alloc_hint_flags": 0, 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: "manifest": { 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: "type": 0 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: "watchers": {}, 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: "shard_versions": [] 2026-02-13T02:28:41.589 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: "shards": [ 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 0, 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: "primary": false, 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [ 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: "missing" 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 1, 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: "primary": true, 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: "object": { 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: "name": "obj10", 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: "nspace": "", 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: "locator": "", 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: "snap": "head", 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: "version": 32 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [ 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: "snapset_inconsistency" 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: "union_shard_errors": [], 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: "selected_object_info": { 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: "oid": { 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: "oid": "obj10", 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: "key": "", 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: "snapid": -2, 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: "hash": 718195851, 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: "max": 0, 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: "pool": 1, 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: "namespace": "" 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: "version": "15'32", 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: "prior_version": "14'10", 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: "last_reqid": "client.4253.0:1", 2026-02-13T02:28:41.590 INFO:tasks.workunit.client.0.trial112.stdout: "user_version": 32, 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: "mtime": "2026-02-13T02:28:16.585927+0000", 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: "local_mtime": "2026-02-13T02:28:16.586214+0000", 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: "lost": 0, 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: "flags": [ 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: "dirty", 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest" 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_seq": 0, 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_size": 0, 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest": "0xaa2f2450", 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: "omap_digest": "0xffffffff", 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: "expected_object_size": 0, 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: "expected_write_size": 0, 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: "alloc_hint_flags": 0, 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: "manifest": { 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: "type": 0 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: "watchers": {}, 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: "shard_versions": [] 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: "shards": [ 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 0, 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: "primary": false, 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: "snapset": { 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 1, 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: "clones": [ 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: "snap": 1, 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: "size": 1032, 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: "overlap": "????", 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: "snaps": [ 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: 1 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.591 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 1, 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: "primary": true, 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: "snapset": { 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 1, 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: "clones": [ 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: "snap": 1, 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: "size": 1032, 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: "overlap": "[]", 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: "snaps": [ 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: 1 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: "object": { 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: "name": "obj11", 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: "nspace": "", 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: "locator": "", 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: "snap": "head", 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: "version": 34 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [ 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: "snapset_inconsistency" 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: "union_shard_errors": [], 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: "selected_object_info": { 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: "oid": { 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: "oid": "obj11", 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: "key": "", 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: "snapid": -2, 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: "hash": 693400951, 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: "max": 0, 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: "pool": 1, 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: "namespace": "" 2026-02-13T02:28:41.592 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: "version": "15'34", 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: "prior_version": "14'11", 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: "last_reqid": "client.4256.0:1", 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: "user_version": 34, 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: "mtime": "2026-02-13T02:28:16.614089+0000", 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: "local_mtime": "2026-02-13T02:28:16.614469+0000", 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: "lost": 0, 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: "flags": [ 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: "dirty", 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest" 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_seq": 0, 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_size": 0, 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest": "0xaa2f2450", 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: "omap_digest": "0xffffffff", 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: "expected_object_size": 0, 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: "expected_write_size": 0, 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: "alloc_hint_flags": 0, 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: "manifest": { 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: "type": 0 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: "watchers": {}, 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: "shard_versions": [] 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: "shards": [ 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 0, 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: "primary": false, 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: "snapset": { 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 1, 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: "clones": [] 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 1, 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: "primary": true, 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: "snapset": { 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 1, 2026-02-13T02:28:41.593 INFO:tasks.workunit.client.0.trial112.stdout: "clones": [ 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: "snap": 1, 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: "size": 1032, 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: "overlap": "[]", 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: "snaps": [ 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: 1 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: "object": { 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: "name": "obj13", 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: "nspace": "", 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: "locator": "", 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: "snap": "head", 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: "version": 38 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [ 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: "snapset_inconsistency" 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: "union_shard_errors": [], 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: "selected_object_info": { 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: "oid": { 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: "oid": "obj13", 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: "key": "", 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: "snapid": -2, 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: "hash": 2087409765, 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: "max": 0, 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: "pool": 1, 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: "namespace": "" 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: "version": "15'38", 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: "prior_version": "14'13", 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: "last_reqid": "client.4262.0:1", 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: "user_version": 38, 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: "mtime": "2026-02-13T02:28:16.672719+0000", 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: "local_mtime": "2026-02-13T02:28:16.673116+0000", 2026-02-13T02:28:41.594 INFO:tasks.workunit.client.0.trial112.stdout: "lost": 0, 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: "flags": [ 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: "dirty", 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest" 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_seq": 0, 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_size": 0, 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest": "0xaa2f2450", 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: "omap_digest": "0xffffffff", 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: "expected_object_size": 0, 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: "expected_write_size": 0, 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: "alloc_hint_flags": 0, 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: "manifest": { 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: "type": 0 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: "watchers": {}, 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: "shard_versions": [] 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: "shards": [ 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 0, 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: "primary": false, 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: "snapset": { 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 1, 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: "clones": [ 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: "snap": 1, 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: "size": 1032, 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: "overlap": "[]" 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 1, 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: "primary": true, 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: "snapset": { 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 1, 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: "clones": [ 2026-02-13T02:28:41.595 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: "snap": 1, 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: "size": 1032, 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: "overlap": "[]", 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: "snaps": [ 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: 1 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: "object": { 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: "name": "obj14", 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: "nspace": "", 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: "locator": "", 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: "snap": "head", 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: "version": 40 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [ 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: "snapset_inconsistency" 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: "union_shard_errors": [], 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: "selected_object_info": { 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: "oid": { 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: "oid": "obj14", 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: "key": "", 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: "snapid": -2, 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: "hash": 2484217095, 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: "max": 0, 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: "pool": 1, 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: "namespace": "" 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: "version": "15'40", 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: "prior_version": "14'14", 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: "last_reqid": "client.4265.0:1", 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: "user_version": 40, 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: "mtime": "2026-02-13T02:28:16.699202+0000", 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: "local_mtime": "2026-02-13T02:28:16.699534+0000", 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: "lost": 0, 2026-02-13T02:28:41.596 INFO:tasks.workunit.client.0.trial112.stdout: "flags": [ 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: "dirty", 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest" 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_seq": 0, 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_size": 0, 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest": "0xaa2f2450", 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: "omap_digest": "0xffffffff", 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: "expected_object_size": 0, 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: "expected_write_size": 0, 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: "alloc_hint_flags": 0, 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: "manifest": { 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: "type": 0 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: "watchers": {}, 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: "shard_versions": [] 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: "shards": [ 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 0, 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: "primary": false, 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: "snapset": { 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 1, 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: "clones": [ 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: "snap": 1, 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: "size": 1033, 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: "overlap": "[]", 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: "snaps": [ 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: 1 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 1, 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: "primary": true, 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: "snapset": { 2026-02-13T02:28:41.597 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 1, 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: "clones": [ 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: "snap": 1, 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: "size": 1032, 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: "overlap": "[]", 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: "snaps": [ 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: 1 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: "object": { 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: "name": "obj15", 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: "nspace": "", 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: "locator": "", 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: "snap": "head", 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: "version": 15 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: "union_shard_errors": [ 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: "snapset_corrupted" 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: "selected_object_info": { 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: "oid": { 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: "oid": "obj15", 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: "key": "", 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: "snapid": -2, 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: "hash": 612772309, 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: "max": 0, 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: "pool": 1, 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: "namespace": "" 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: "version": "14'15", 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: "prior_version": "0'0", 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: "last_reqid": "client.4217.0:1", 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: "user_version": 15, 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: "size": 1032, 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: "mtime": "2026-02-13T02:28:15.762572+0000", 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: "local_mtime": "2026-02-13T02:28:15.762906+0000", 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: "lost": 0, 2026-02-13T02:28:41.598 INFO:tasks.workunit.client.0.trial112.stdout: "flags": [ 2026-02-13T02:28:41.599 INFO:tasks.workunit.client.0.trial112.stdout: "dirty", 2026-02-13T02:28:41.599 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest" 2026-02-13T02:28:41.599 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:28:41.599 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_seq": 0, 2026-02-13T02:28:41.599 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_size": 0, 2026-02-13T02:28:41.599 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest": "0xd5ca59ae", 2026-02-13T02:28:41.599 INFO:tasks.workunit.client.0.trial112.stdout: "omap_digest": "0xffffffff", 2026-02-13T02:28:41.599 INFO:tasks.workunit.client.0.trial112.stdout: "expected_object_size": 0, 2026-02-13T02:28:41.599 INFO:tasks.workunit.client.0.trial112.stdout: "expected_write_size": 0, 2026-02-13T02:28:41.599 INFO:tasks.workunit.client.0.trial112.stdout: "alloc_hint_flags": 0, 2026-02-13T02:28:41.599 INFO:tasks.workunit.client.0.trial112.stdout: "manifest": { 2026-02-13T02:28:41.599 INFO:tasks.workunit.client.0.trial112.stdout: "type": 0 2026-02-13T02:28:41.599 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.599 INFO:tasks.workunit.client.0.trial112.stdout: "watchers": {}, 2026-02-13T02:28:41.599 INFO:tasks.workunit.client.0.trial112.stdout: "shard_versions": [] 2026-02-13T02:28:41.599 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.599 INFO:tasks.workunit.client.0.trial112.stdout: "shards": [ 2026-02-13T02:28:41.599 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.599 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 0, 2026-02-13T02:28:41.599 INFO:tasks.workunit.client.0.trial112.stdout: "primary": false, 2026-02-13T02:28:41.599 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [ 2026-02-13T02:28:41.599 INFO:tasks.workunit.client.0.trial112.stdout: "snapset_corrupted" 2026-02-13T02:28:41.599 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:28:41.599 INFO:tasks.workunit.client.0.trial112.stdout: "size": 1032, 2026-02-13T02:28:41.599 INFO:tasks.workunit.client.0.trial112.stdout: "snapset": "Z2FyYmFnZQo=" 2026-02-13T02:28:41.599 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.599 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.599 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 1, 2026-02-13T02:28:41.599 INFO:tasks.workunit.client.0.trial112.stdout: "primary": true, 2026-02-13T02:28:41.599 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:28:41.599 INFO:tasks.workunit.client.0.trial112.stdout: "size": 1032, 2026-02-13T02:28:41.599 INFO:tasks.workunit.client.0.trial112.stdout: "snapset": { 2026-02-13T02:28:41.599 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 0, 2026-02-13T02:28:41.599 INFO:tasks.workunit.client.0.trial112.stdout: "clones": [] 2026-02-13T02:28:41.599 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.599 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.599 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.599 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.599 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: "object": { 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: "name": "obj2", 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: "nspace": "", 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: "locator": "", 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: "snap": "head", 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: "version": 56 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: "union_shard_errors": [ 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: "snapset_missing" 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: "selected_object_info": { 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: "oid": { 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: "oid": "obj2", 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: "key": "", 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: "snapid": -2, 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: "hash": 1058988552, 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: "max": 0, 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: "pool": 1, 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: "namespace": "" 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: "version": "21'56", 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: "prior_version": "18'48", 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: "last_reqid": "client.4307.0:1", 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: "user_version": 56, 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: "size": 0, 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: "mtime": "2026-02-13T02:28:17.365081+0000", 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: "local_mtime": "2026-02-13T02:28:17.365507+0000", 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: "lost": 0, 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: "flags": [ 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: "whiteout", 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: "dirty" 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_seq": 0, 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_size": 0, 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest": "0xffffffff", 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: "omap_digest": "0xffffffff", 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: "expected_object_size": 0, 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: "expected_write_size": 0, 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: "alloc_hint_flags": 0, 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: "manifest": { 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: "type": 0 2026-02-13T02:28:41.600 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: "watchers": {}, 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: "shard_versions": [] 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: "shards": [ 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 0, 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: "primary": false, 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [ 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: "snapset_missing" 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: "size": 0 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 1, 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: "primary": true, 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: "size": 0, 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: "snapset": { 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 7, 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: "clones": [ 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: "snap": 4, 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: "size": 1032, 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: "overlap": "[]", 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: "snaps": [ 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: 4, 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: 3, 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: 2, 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: 1 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: "snap": 7, 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: "size": 1024, 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: "overlap": "[]", 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: "snaps": [ 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: 7, 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: 6, 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: 5 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.601 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: "object": { 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: "name": "obj3", 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: "nspace": "", 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: "locator": "", 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: "snap": "head", 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: "version": 44 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [ 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: "size_mismatch" 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: "union_shard_errors": [ 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: "size_mismatch_info", 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: "obj_size_info_mismatch" 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: "selected_object_info": { 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: "oid": { 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: "oid": "obj3", 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: "key": "", 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: "snapid": -2, 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: "hash": 1643547569, 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: "max": 0, 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: "pool": 1, 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: "namespace": "" 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: "version": "17'44", 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: "prior_version": "15'22", 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: "last_reqid": "client.4277.0:1", 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: "user_version": 44, 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: "size": 768, 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: "mtime": "2026-02-13T02:28:16.904095+0000", 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: "local_mtime": "2026-02-13T02:28:16.905131+0000", 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: "lost": 0, 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: "flags": [ 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: "dirty", 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest" 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_seq": 0, 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_size": 0, 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest": "0xdb556c60", 2026-02-13T02:28:41.602 INFO:tasks.workunit.client.0.trial112.stdout: "omap_digest": "0xffffffff", 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: "expected_object_size": 0, 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: "expected_write_size": 0, 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: "alloc_hint_flags": 0, 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: "manifest": { 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: "type": 0 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: "watchers": {}, 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: "shard_versions": [] 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: "shards": [ 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 0, 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: "primary": false, 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [ 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: "size_mismatch_info", 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: "obj_size_info_mismatch" 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: "size": 3840, 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: "object_info": { 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: "oid": { 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: "oid": "obj3", 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: "key": "", 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: "snapid": -2, 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: "hash": 1643547569, 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: "max": 0, 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: "pool": 1, 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: "namespace": "" 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: "version": "17'44", 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: "prior_version": "15'22", 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: "last_reqid": "client.4277.0:1", 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: "user_version": 44, 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: "size": 768, 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: "mtime": "2026-02-13T02:28:16.904095+0000", 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: "local_mtime": "2026-02-13T02:28:16.905131+0000", 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: "lost": 0, 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: "flags": [ 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: "dirty", 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest" 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_seq": 0, 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_size": 0, 2026-02-13T02:28:41.603 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest": "0xdb556c60", 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: "omap_digest": "0xffffffff", 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: "expected_object_size": 0, 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: "expected_write_size": 0, 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: "alloc_hint_flags": 0, 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: "manifest": { 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: "type": 0 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: "watchers": {}, 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: "shard_versions": [] 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 1, 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: "primary": true, 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: "size": 768 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: "object": { 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: "name": "obj6", 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: "nspace": "", 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: "locator": "", 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: "snap": "head", 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: "version": 24 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [ 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: "snapset_inconsistency" 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: "union_shard_errors": [], 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: "selected_object_info": { 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: "oid": { 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: "oid": "obj6", 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: "key": "", 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: "snapid": -2, 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: "hash": 2202164420, 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: "max": 0, 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: "pool": 1, 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: "namespace": "" 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: "version": "15'24", 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: "prior_version": "14'6", 2026-02-13T02:28:41.604 INFO:tasks.workunit.client.0.trial112.stdout: "last_reqid": "client.4241.0:1", 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: "user_version": 24, 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: "mtime": "2026-02-13T02:28:16.477321+0000", 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: "local_mtime": "2026-02-13T02:28:16.477770+0000", 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: "lost": 0, 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: "flags": [ 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: "dirty", 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest" 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_seq": 0, 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_size": 0, 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest": "0xaa2f2450", 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: "omap_digest": "0xffffffff", 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: "expected_object_size": 0, 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: "expected_write_size": 0, 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: "alloc_hint_flags": 0, 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: "manifest": { 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: "type": 0 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: "watchers": {}, 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: "shard_versions": [] 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: "shards": [ 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 0, 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: "primary": false, 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: "snapset": { 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 1, 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: "clones": [] 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 1, 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: "primary": true, 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: "snapset": { 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 1, 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: "clones": [ 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: "snap": 1, 2026-02-13T02:28:41.605 INFO:tasks.workunit.client.0.trial112.stdout: "size": 1032, 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: "overlap": "[]", 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: "snaps": [ 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: 1 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: "object": { 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: "name": "obj7", 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: "nspace": "", 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: "locator": "", 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: "snap": "head", 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: "version": 26 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [ 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: "snapset_inconsistency" 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: "union_shard_errors": [], 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: "selected_object_info": { 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: "oid": { 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: "oid": "obj7", 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: "key": "", 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: "snapid": -2, 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: "hash": 1552453721, 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: "max": 0, 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: "pool": 1, 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: "namespace": "" 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: "version": "15'26", 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: "prior_version": "14'7", 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: "last_reqid": "client.4244.0:1", 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: "user_version": 26, 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: "mtime": "2026-02-13T02:28:16.506359+0000", 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: "local_mtime": "2026-02-13T02:28:16.506704+0000", 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: "lost": 0, 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: "flags": [ 2026-02-13T02:28:41.606 INFO:tasks.workunit.client.0.trial112.stdout: "dirty", 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest" 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_seq": 0, 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_size": 0, 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest": "0xaa2f2450", 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: "omap_digest": "0xffffffff", 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: "expected_object_size": 0, 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: "expected_write_size": 0, 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: "alloc_hint_flags": 0, 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: "manifest": { 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: "type": 0 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: "watchers": {}, 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: "shard_versions": [] 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: "shards": [ 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 0, 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: "primary": false, 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: "snapset": { 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 0, 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: "clones": [] 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 1, 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: "primary": true, 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: "snapset": { 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 1, 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: "clones": [ 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: "snap": 1, 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: "size": 1032, 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: "overlap": "[]", 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: "snaps": [ 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: 1 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.607 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: "object": { 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: "name": "obj8", 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: "nspace": "", 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: "locator": "", 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: "snap": "head", 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: "version": 28 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [ 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: "snapset_inconsistency" 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: "union_shard_errors": [], 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: "selected_object_info": { 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: "oid": { 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: "oid": "obj8", 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: "key": "", 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: "snapid": -2, 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: "hash": 2381834917, 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: "max": 0, 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: "pool": 1, 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: "namespace": "" 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: "version": "15'28", 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: "prior_version": "14'8", 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: "last_reqid": "client.4247.0:1", 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: "user_version": 28, 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: "mtime": "2026-02-13T02:28:16.534811+0000", 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: "local_mtime": "2026-02-13T02:28:16.535226+0000", 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: "lost": 0, 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: "flags": [ 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: "dirty", 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest" 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_seq": 0, 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_size": 0, 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest": "0xaa2f2450", 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: "omap_digest": "0xffffffff", 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: "expected_object_size": 0, 2026-02-13T02:28:41.608 INFO:tasks.workunit.client.0.trial112.stdout: "expected_write_size": 0, 2026-02-13T02:28:41.609 INFO:tasks.workunit.client.0.trial112.stdout: "alloc_hint_flags": 0, 2026-02-13T02:28:41.609 INFO:tasks.workunit.client.0.trial112.stdout: "manifest": { 2026-02-13T02:28:41.609 INFO:tasks.workunit.client.0.trial112.stdout: "type": 0 2026-02-13T02:28:41.609 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.609 INFO:tasks.workunit.client.0.trial112.stdout: "watchers": {}, 2026-02-13T02:28:41.609 INFO:tasks.workunit.client.0.trial112.stdout: "shard_versions": [] 2026-02-13T02:28:41.609 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.609 INFO:tasks.workunit.client.0.trial112.stdout: "shards": [ 2026-02-13T02:28:41.609 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.609 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 0, 2026-02-13T02:28:41.609 INFO:tasks.workunit.client.0.trial112.stdout: "primary": false, 2026-02-13T02:28:41.609 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:28:41.609 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:28:41.609 INFO:tasks.workunit.client.0.trial112.stdout: "snapset": { 2026-02-13T02:28:41.609 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 0, 2026-02-13T02:28:41.609 INFO:tasks.workunit.client.0.trial112.stdout: "clones": [ 2026-02-13T02:28:41.609 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.609 INFO:tasks.workunit.client.0.trial112.stdout: "snap": 1, 2026-02-13T02:28:41.609 INFO:tasks.workunit.client.0.trial112.stdout: "size": 1032, 2026-02-13T02:28:41.609 INFO:tasks.workunit.client.0.trial112.stdout: "overlap": "[]", 2026-02-13T02:28:41.609 INFO:tasks.workunit.client.0.trial112.stdout: "snaps": [ 2026-02-13T02:28:41.609 INFO:tasks.workunit.client.0.trial112.stdout: 1 2026-02-13T02:28:41.609 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.609 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.609 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.609 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.609 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.609 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.609 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 1, 2026-02-13T02:28:41.609 INFO:tasks.workunit.client.0.trial112.stdout: "primary": true, 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: "snapset": { 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 1, 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: "clones": [ 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: "snap": 1, 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: "size": 1032, 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: "overlap": "[]", 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: "snaps": [ 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: 1 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: "object": { 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: "name": "obj9", 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: "nspace": "", 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: "locator": "", 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: "snap": "head", 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: "version": 30 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [ 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: "snapset_inconsistency" 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: "union_shard_errors": [], 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: "selected_object_info": { 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: "oid": { 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: "oid": "obj9", 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: "key": "", 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: "snapid": -2, 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: "hash": 3833113727, 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: "max": 0, 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: "pool": 1, 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: "namespace": "" 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: "version": "15'30", 2026-02-13T02:28:41.610 INFO:tasks.workunit.client.0.trial112.stdout: "prior_version": "14'9", 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: "last_reqid": "client.4250.0:1", 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: "user_version": 30, 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: "mtime": "2026-02-13T02:28:16.564306+0000", 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: "local_mtime": "2026-02-13T02:28:16.564578+0000", 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: "lost": 0, 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: "flags": [ 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: "dirty", 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest" 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_seq": 0, 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: "truncate_size": 0, 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: "data_digest": "0xaa2f2450", 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: "omap_digest": "0xffffffff", 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: "expected_object_size": 0, 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: "expected_write_size": 0, 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: "alloc_hint_flags": 0, 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: "manifest": { 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: "type": 0 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: "watchers": {}, 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: "shard_versions": [] 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: "shards": [ 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 0, 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: "primary": false, 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: "snapset": { 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 1, 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: "clones": [ 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: "snap": 1, 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: "size": "????", 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: "overlap": "[]", 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: "snaps": [ 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: 1 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.611 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:28:41.612 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.612 INFO:tasks.workunit.client.0.trial112.stdout: "osd": 1, 2026-02-13T02:28:41.612 INFO:tasks.workunit.client.0.trial112.stdout: "primary": true, 2026-02-13T02:28:41.612 INFO:tasks.workunit.client.0.trial112.stdout: "errors": [], 2026-02-13T02:28:41.612 INFO:tasks.workunit.client.0.trial112.stdout: "size": 256, 2026-02-13T02:28:41.612 INFO:tasks.workunit.client.0.trial112.stdout: "snapset": { 2026-02-13T02:28:41.612 INFO:tasks.workunit.client.0.trial112.stdout: "seq": 1, 2026-02-13T02:28:41.612 INFO:tasks.workunit.client.0.trial112.stdout: "clones": [ 2026-02-13T02:28:41.612 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:28:41.612 INFO:tasks.workunit.client.0.trial112.stdout: "snap": 1, 2026-02-13T02:28:41.612 INFO:tasks.workunit.client.0.trial112.stdout: "size": 1032, 2026-02-13T02:28:41.612 INFO:tasks.workunit.client.0.trial112.stdout: "overlap": "[]", 2026-02-13T02:28:41.612 INFO:tasks.workunit.client.0.trial112.stdout: "snaps": [ 2026-02-13T02:28:41.612 INFO:tasks.workunit.client.0.trial112.stdout: 1 2026-02-13T02:28:41.612 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.612 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.612 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.612 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.612 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.612 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.612 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:28:41.612 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:28:41.612 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:28:41.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:797: _scrub_snaps_multi: rados list-inconsistent-snapset 1.0 2026-02-13T02:28:41.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:800: _scrub_snaps_multi: '[' replica = replica ']' 2026-02-13T02:28:41.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:802: _scrub_snaps_multi: scruberrors=20 2026-02-13T02:28:41.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:803: _scrub_snaps_multi: jq .inconsistents 2026-02-13T02:28:41.618 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:803: _scrub_snaps_multi: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2026-02-13T02:28:41.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1066: _scrub_snaps_multi: jq .inconsistents td/osd-scrub-snaps/json 2026-02-13T02:28:41.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1066: _scrub_snaps_multi: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2026-02-13T02:28:41.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1067: _scrub_snaps_multi: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2026-02-13T02:28:41.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2513: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2026-02-13T02:28:41.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1068: _scrub_snaps_multi: test no = yes 2026-02-13T02:28:41.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1073: _scrub_snaps_multi: test '' = yes 2026-02-13T02:28:41.654 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: find td/osd-scrub-snaps 2026-02-13T02:28:41.654 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: grep 'osd[^/]*\.pid' 2026-02-13T02:28:41.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: pidfiles='td/osd-scrub-snaps/osd.1.pid 2026-02-13T02:28:41.656 INFO:tasks.workunit.client.0.trial112.stderr:td/osd-scrub-snaps/osd.0.pid' 2026-02-13T02:28:41.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1079: _scrub_snaps_multi: pids= 2026-02-13T02:28:41.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1080: _scrub_snaps_multi: for pidfile in ${pidfiles} 2026-02-13T02:28:41.657 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: cat td/osd-scrub-snaps/osd.1.pid 2026-02-13T02:28:41.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: pids+='507343 ' 2026-02-13T02:28:41.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1080: _scrub_snaps_multi: for pidfile in ${pidfiles} 2026-02-13T02:28:41.658 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: cat td/osd-scrub-snaps/osd.0.pid 2026-02-13T02:28:41.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: pids+='506801 ' 2026-02-13T02:28:41.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1085: _scrub_snaps_multi: ERRORS=0 2026-02-13T02:28:41.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1089: _scrub_snaps_multi: '[' replica = primary ']' 2026-02-13T02:28:41.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1111: _scrub_snaps_multi: ceph pg dump pgs 2026-02-13T02:28:41.811 INFO:tasks.workunit.client.0.trial112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-13T02:28:41.811 INFO:tasks.workunit.client.0.trial112.stdout:1.0 36 0 0 0 0 24448 0 0 56 0 56 active+clean+inconsistent 2026-02-13T02:28:34.625443+0000 21'56 30:107 [1,0] 1 [1,0] 1 21'56 2026-02-13T02:28:34.625388+0000 0'0 2026-02-13T02:28:10.433137+0000 0 1 periodic scrub scheduled @ 2026-02-14T11:37:03.760179+0000 36 0 2026-02-13T02:28:41.811 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T02:28:41.811 INFO:tasks.workunit.client.0.trial112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-13T02:28:41.811 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:28:41.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1113: _scrub_snaps_multi: for pid in $pids 2026-02-13T02:28:41.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1115: _scrub_snaps_multi: kill -0 507343 2026-02-13T02:28:41.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1113: _scrub_snaps_multi: for pid in $pids 2026-02-13T02:28:41.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1115: _scrub_snaps_multi: kill -0 506801 2026-02-13T02:28:41.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1122: _scrub_snaps_multi: kill_daemons td/osd-scrub-snaps 2026-02-13T02:28:41.820 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:28:41.820 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:28:41.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:28:41.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:28:41.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:28:47.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:28:47.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1124: _scrub_snaps_multi: declare -a err_strings 2026-02-13T02:28:47.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1125: _scrub_snaps_multi: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj4:7 : missing' 2026-02-13T02:28:47.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1126: _scrub_snaps_multi: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj3:head : size 3840 != size 768 from auth oi' 2026-02-13T02:28:47.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1127: _scrub_snaps_multi: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:1 : missing' 2026-02-13T02:28:47.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1128: _scrub_snaps_multi: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:2 : missing' 2026-02-13T02:28:47.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1129: _scrub_snaps_multi: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj5:4 : size 4608 != size 512 from auth oi' 2026-02-13T02:28:47.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1130: _scrub_snaps_multi: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid .*:::obj5:7 : failed to pick suitable object info' 2026-02-13T02:28:47.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj1:head : missing' 2026-02-13T02:28:47.138 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1132: _scrub_snaps_multi: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' 2026-02-13T02:28:47.138 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-13T02:28:47.138 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj4:7 : missing' td/osd-scrub-snaps/osd.1.log 2026-02-13T02:28:47.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-13T02:28:47.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj3:head : size 3840 != size 768 from auth oi' td/osd-scrub-snaps/osd.1.log 2026-02-13T02:28:47.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-13T02:28:47.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:1 : missing' td/osd-scrub-snaps/osd.1.log 2026-02-13T02:28:47.144 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-13T02:28:47.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:2 : missing' td/osd-scrub-snaps/osd.1.log 2026-02-13T02:28:47.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-13T02:28:47.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj5:4 : size 4608 != size 512 from auth oi' td/osd-scrub-snaps/osd.1.log 2026-02-13T02:28:47.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-13T02:28:47.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid .*:::obj5:7 : failed to pick suitable object info' td/osd-scrub-snaps/osd.1.log 2026-02-13T02:28:47.152 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-13T02:28:47.152 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj1:head : missing' td/osd-scrub-snaps/osd.1.log 2026-02-13T02:28:47.155 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-13T02:28:47.155 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' td/osd-scrub-snaps/osd.1.log 2026-02-13T02:28:47.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1144: _scrub_snaps_multi: declare -a rep_err_strings 2026-02-13T02:28:47.158 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: eval echo '$replica' 2026-02-13T02:28:47.158 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: echo 0 2026-02-13T02:28:47.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: osd=0 2026-02-13T02:28:47.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1146: _scrub_snaps_multi: rep_err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' 2026-02-13T02:28:47.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1147: _scrub_snaps_multi: for err_string in "${rep_err_strings[@]}" 2026-02-13T02:28:47.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1149: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' td/osd-scrub-snaps/osd.0.log 2026-02-13T02:28:47.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1156: _scrub_snaps_multi: '[' 0 '!=' 0 ']' 2026-02-13T02:28:47.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1162: _scrub_snaps_multi: echo 'TEST PASSED' 2026-02-13T02:28:47.161 INFO:tasks.workunit.client.0.trial112.stdout:TEST PASSED 2026-02-13T02:28:47.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1163: _scrub_snaps_multi: return 0 2026-02-13T02:28:47.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1171: TEST_scrub_snaps_replica: err=0 2026-02-13T02:28:47.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1172: TEST_scrub_snaps_replica: CEPH_ARGS='--fsid=b2b95964-f5a3-4b20-bd2b-d60f1ae5f8a0 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2026-02-13T02:28:47.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1173: TEST_scrub_snaps_replica: return 0 2026-02-13T02:28:47.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:40: run: teardown td/osd-scrub-snaps 2026-02-13T02:28:47.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2026-02-13T02:28:47.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:28:47.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2026-02-13T02:28:47.162 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:28:47.162 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:28:47.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:28:47.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:28:47.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:28:47.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:28:47.168 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:28:47.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:28:47.170 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:28:47.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:28:47.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:28:47.172 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:28:47.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:28:47.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:28:47.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:28:47.174 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:28:47.175 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:28:47.176 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:28:47.176 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2026-02-13T02:28:47.185 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:28:47.185 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:28:47.185 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.453125 2026-02-13T02:28:47.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.453125 2026-02-13T02:28:47.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:28:47.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:28:47.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: main: code=0 2026-02-13T02:28:47.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: teardown td/osd-scrub-snaps 0 2026-02-13T02:28:47.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2026-02-13T02:28:47.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-13T02:28:47.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2026-02-13T02:28:47.187 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:28:47.187 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:28:47.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:28:47.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:28:47.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:28:47.190 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:28:47.191 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:28:47.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:28:47.192 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:28:47.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:28:47.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:28:47.194 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:28:47.195 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:28:47.195 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:28:47.196 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:28:47.196 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:28:47.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:28:47.198 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-13T02:28:47.198 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2026-02-13T02:28:47.199 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:28:47.199 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:28:47.199 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.453125 2026-02-13T02:28:47.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.453125 2026-02-13T02:28:47.202 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:28:47.202 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:28:47.202 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: main: return 0 2026-02-13T02:28:47.203 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-13T02:28:47.203 DEBUG:teuthology.orchestra.run.trial112:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-13T02:28:47.256 INFO:tasks.workunit:Running workunit scrub/osd-scrub-test.sh... 2026-02-13T02:28:47.256 DEBUG:teuthology.orchestra.run.trial112:workunit test scrub/osd-scrub-test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh 2026-02-13T02:28:47.308 INFO:tasks.workunit.client.0.trial112.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-13T02:28:47.315 INFO:tasks.workunit.client.0.trial112.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-13T02:28:47.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-13T02:28:47.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-13T02:28:47.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export PYTHONWARNINGS=ignore 2026-02-13T02:28:47.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: PYTHONWARNINGS=ignore 2026-02-13T02:28:47.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: export CEPH_CONF=/dev/null 2026-02-13T02:28:47.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: CEPH_CONF=/dev/null 2026-02-13T02:28:47.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: main: unset CEPH_ARGS 2026-02-13T02:28:47.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: local code 2026-02-13T02:28:47.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: run td/osd-scrub-test 2026-02-13T02:28:47.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:21: run: local dir=td/osd-scrub-test 2026-02-13T02:28:47.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:22: run: shift 2026-02-13T02:28:47.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:24: run: export CEPH_MON=127.0.0.1:7138 2026-02-13T02:28:47.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:24: run: CEPH_MON=127.0.0.1:7138 2026-02-13T02:28:47.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:25: run: export CEPH_ARGS 2026-02-13T02:28:47.317 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:26: run: uuidgen 2026-02-13T02:28:47.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:26: run: CEPH_ARGS+='--fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none ' 2026-02-13T02:28:47.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7138 ' 2026-02-13T02:28:47.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:29: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2026-02-13T02:28:47.318 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:30: run: set 2026-02-13T02:28:47.318 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-13T02:28:47.319 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:30: run: local 'funcs=TEST_abort_periodic_for_operator 2026-02-13T02:28:47.320 INFO:tasks.workunit.client.0.trial112.stderr:TEST_deep_scrub_abort 2026-02-13T02:28:47.320 INFO:tasks.workunit.client.0.trial112.stderr:TEST_dump_scrub_schedule 2026-02-13T02:28:47.320 INFO:tasks.workunit.client.0.trial112.stderr:TEST_interval_changes 2026-02-13T02:28:47.320 INFO:tasks.workunit.client.0.trial112.stderr:TEST_just_deep_scrubs 2026-02-13T02:28:47.320 INFO:tasks.workunit.client.0.trial112.stderr:TEST_pg_dump_objects_scrubbed 2026-02-13T02:28:47.320 INFO:tasks.workunit.client.0.trial112.stderr:TEST_scrub_abort 2026-02-13T02:28:47.320 INFO:tasks.workunit.client.0.trial112.stderr:TEST_scrub_permit_time 2026-02-13T02:28:47.320 INFO:tasks.workunit.client.0.trial112.stderr:TEST_scrub_test' 2026-02-13T02:28:47.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2026-02-13T02:28:47.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_abort_periodic_for_operator -------------------' 2026-02-13T02:28:47.321 INFO:tasks.workunit.client.0.trial112.stdout:-------------- Prepare Test TEST_abort_periodic_for_operator ------------------- 2026-02-13T02:28:47.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2026-02-13T02:28:47.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2026-02-13T02:28:47.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2026-02-13T02:28:47.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-13T02:28:47.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:28:47.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-13T02:28:47.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:28:47.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:28:47.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:28:47.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:28:47.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:28:47.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:28:47.324 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:28:47.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:28:47.324 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:28:47.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:28:47.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:28:47.325 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:28:47.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:28:47.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:28:47.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:28:47.327 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:28:47.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:28:47.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:28:47.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-13T02:28:47.331 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:28:47.331 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:28:47.331 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:28:47.331 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.508946 2026-02-13T02:28:47.332 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:28:47.332 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:28:47.332 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2026-02-13T02:28:47.334 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T02:28:47.334 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:28:47.334 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:28:47.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.508946 2026-02-13T02:28:47.336 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T02:28:47.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-13T02:28:47.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-13T02:28:47.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T02:28:47.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2026-02-13T02:28:47.337 INFO:tasks.workunit.client.0.trial112.stdout:-------------- Run Test TEST_abort_periodic_for_operator ----------------------- 2026-02-13T02:28:47.339 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_abort_periodic_for_operator -----------------------' 2026-02-13T02:28:47.339 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_abort_periodic_for_operator td/osd-scrub-test 2026-02-13T02:28:47.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:748: TEST_abort_periodic_for_operator: local dir=td/osd-scrub-test 2026-02-13T02:28:47.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:749: TEST_abort_periodic_for_operator: cluster_conf=(['osds_num']='5' ['pgs_in_pool']='16' ['pool_name']='test') 2026-02-13T02:28:47.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:749: TEST_abort_periodic_for_operator: local -A cluster_conf 2026-02-13T02:28:47.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:754: TEST_abort_periodic_for_operator: local extr_dbg=1 2026-02-13T02:28:47.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:756: TEST_abort_periodic_for_operator: standard_scrub_wpq_cluster td/osd-scrub-test cluster_conf 2 2026-02-13T02:28:47.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:302: standard_scrub_wpq_cluster: local dir=td/osd-scrub-test 2026-02-13T02:28:47.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:303: standard_scrub_wpq_cluster: local -n conf=cluster_conf 2026-02-13T02:28:47.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:304: standard_scrub_wpq_cluster: local osd_sleep=2 2026-02-13T02:28:47.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:306: standard_scrub_wpq_cluster: conf['extras']=' --osd_op_queue=wpq --osd_scrub_sleep=2 ' 2026-02-13T02:28:47.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:308: standard_scrub_wpq_cluster: standard_scrub_cluster td/osd-scrub-test conf 2026-02-13T02:28:47.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local dir=td/osd-scrub-test 2026-02-13T02:28:47.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:230: standard_scrub_cluster: local -n args=conf 2026-02-13T02:28:47.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local OSDS=5 2026-02-13T02:28:47.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: local pg_num=16 2026-02-13T02:28:47.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:235: standard_scrub_cluster: local poolname=test 2026-02-13T02:28:47.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:236: standard_scrub_cluster: args['pool_name']=test 2026-02-13T02:28:47.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:238: standard_scrub_cluster: local pool_default_size=3 2026-02-13T02:28:47.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:239: standard_scrub_cluster: args['pool_default_size']=3 2026-02-13T02:28:47.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:241: standard_scrub_cluster: local 'extra_pars= --osd_op_queue=wpq --osd_scrub_sleep=2 ' 2026-02-13T02:28:47.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:242: standard_scrub_cluster: local debug_msg=dbg 2026-02-13T02:28:47.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:245: standard_scrub_cluster: local saved_echo_flag=x 2026-02-13T02:28:47.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:246: standard_scrub_cluster: set +x 2026-02-13T02:28:47.625 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 1cc2d59d-46ee-4155-9013-f68d8fc420cd 2026-02-13T02:28:47.725 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:28:47.753 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:47.746+0000 7f1292dcaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:47.754 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:47.746+0000 7f1292dcaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:47.755 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:47.746+0000 7f1292dcaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:47.756 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:47.750+0000 7f1292dcaa40 -1 bdev(0x55e5bdc76c00 td/osd-scrub-test/0/block) open stat got: (1) Operation not permitted 2026-02-13T02:28:47.756 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:47.750+0000 7f1292dcaa40 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2026-02-13T02:28:48.432 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T02:28:48.525 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T02:28:48.544 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:48.538+0000 7f665322da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:48.545 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:48.538+0000 7f665322da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:48.546 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:48.538+0000 7f665322da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:48.620 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:28:48.945 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:48.938+0000 7f665322da40 -1 Falling back to public interface 2026-02-13T02:28:49.556 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:49.550+0000 7f665322da40 -1 osd.0 0 log_to_monitors true 2026-02-13T02:28:49.714 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:28:50.814 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:28:50.975 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2584913244,v1:127.0.0.1:6801/2584913244] [v2:127.0.0.1:6802/2584913244,v1:127.0.0.1:6803/2584913244] exists,up 1cc2d59d-46ee-4155-9013-f68d8fc420cd 2026-02-13T02:28:50.980 INFO:tasks.workunit.client.0.trial112.stdout:add osd1 78eed3ca-2827-42cc-a045-8bf5fb348c3b 2026-02-13T02:28:51.153 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:28:51.183 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:51.174+0000 7f8937f2aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:51.185 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:51.178+0000 7f8937f2aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:51.186 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:51.178+0000 7f8937f2aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:51.186 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:51.178+0000 7f8937f2aa40 -1 bdev(0x55a34e4dcc00 td/osd-scrub-test/1/block) open stat got: (1) Operation not permitted 2026-02-13T02:28:51.186 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:51.178+0000 7f8937f2aa40 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2026-02-13T02:28:51.832 INFO:tasks.workunit.client.0.trial112.stdout:adding osd1 key to auth repository 2026-02-13T02:28:52.035 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T02:28:52.066 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:52.058+0000 7feba7780a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:52.067 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:52.058+0000 7feba7780a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:52.068 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:52.062+0000 7feba7780a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:52.197 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:28:52.474 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:52.466+0000 7feba7780a40 -1 Falling back to public interface 2026-02-13T02:28:53.054 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:53.046+0000 7feba7780a40 -1 osd.1 0 log_to_monitors true 2026-02-13T02:28:53.356 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:28:53.517 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/353044745,v1:127.0.0.1:6811/353044745] [v2:127.0.0.1:6812/353044745,v1:127.0.0.1:6813/353044745] exists,up 78eed3ca-2827-42cc-a045-8bf5fb348c3b 2026-02-13T02:28:53.522 INFO:tasks.workunit.client.0.trial112.stdout:add osd2 d71a4976-fecf-4a1c-a8af-1bfdb96f2593 2026-02-13T02:28:53.696 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:28:53.728 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:53.718+0000 7fca3f877a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:53.729 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:53.722+0000 7fca3f877a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:53.730 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:53.722+0000 7fca3f877a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:53.731 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:53.722+0000 7fca3f877a40 -1 bdev(0x55f390e9ac00 td/osd-scrub-test/2/block) open stat got: (1) Operation not permitted 2026-02-13T02:28:53.731 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:53.722+0000 7fca3f877a40 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2026-02-13T02:28:54.351 INFO:tasks.workunit.client.0.trial112.stdout:adding osd2 key to auth repository 2026-02-13T02:28:54.555 INFO:tasks.workunit.client.0.trial112.stdout:start osd.2 2026-02-13T02:28:54.574 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:54.566+0000 7ff8cb344a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:54.575 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:54.566+0000 7ff8cb344a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:54.576 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:54.570+0000 7ff8cb344a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:54.713 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:28:54.981 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:54.974+0000 7ff8cb344a40 -1 Falling back to public interface 2026-02-13T02:28:55.543 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:55.534+0000 7ff8cb344a40 -1 osd.2 0 log_to_monitors true 2026-02-13T02:28:55.878 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:28:56.042 INFO:tasks.workunit.client.0.trial112.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4249931764,v1:127.0.0.1:6819/4249931764] [v2:127.0.0.1:6820/4249931764,v1:127.0.0.1:6821/4249931764] exists,up d71a4976-fecf-4a1c-a8af-1bfdb96f2593 2026-02-13T02:28:56.047 INFO:tasks.workunit.client.0.trial112.stdout:add osd3 b59b78c0-f64b-4acc-b9f8-7938277b2036 2026-02-13T02:28:56.221 INFO:tasks.workunit.client.0.trial112.stdout:3 2026-02-13T02:28:56.251 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:56.242+0000 7f26a8f5da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:56.253 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:56.246+0000 7f26a8f5da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:56.254 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:56.246+0000 7f26a8f5da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:56.254 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:56.246+0000 7f26a8f5da40 -1 bdev(0x556351c02c00 td/osd-scrub-test/3/block) open stat got: (1) Operation not permitted 2026-02-13T02:28:56.255 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:56.246+0000 7f26a8f5da40 -1 bluestore(td/osd-scrub-test/3) _read_fsid unparsable uuid 2026-02-13T02:28:56.884 INFO:tasks.workunit.client.0.trial112.stdout:adding osd3 key to auth repository 2026-02-13T02:28:57.076 INFO:tasks.workunit.client.0.trial112.stdout:start osd.3 2026-02-13T02:28:57.105 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:57.098+0000 7ff3a3abaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:57.107 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:57.098+0000 7ff3a3abaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:57.108 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:57.098+0000 7ff3a3abaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:57.237 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:28:57.545 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:57.538+0000 7ff3a3abaa40 -1 Falling back to public interface 2026-02-13T02:28:58.166 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:58.158+0000 7ff3a3abaa40 -1 osd.3 0 log_to_monitors true 2026-02-13T02:28:58.393 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:28:58.550 INFO:tasks.workunit.client.0.trial112.stdout:osd.3 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1960623339,v1:127.0.0.1:6827/1960623339] [v2:127.0.0.1:6828/1960623339,v1:127.0.0.1:6829/1960623339] exists,up b59b78c0-f64b-4acc-b9f8-7938277b2036 2026-02-13T02:28:58.555 INFO:tasks.workunit.client.0.trial112.stdout:add osd4 2d981b71-1b7b-42f4-b70b-4ef3cc129870 2026-02-13T02:28:58.734 INFO:tasks.workunit.client.0.trial112.stdout:4 2026-02-13T02:28:58.765 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:58.758+0000 7fbd50310a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:58.766 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:58.758+0000 7fbd50310a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:58.767 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:58.758+0000 7fbd50310a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:58.768 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:58.762+0000 7fbd50310a40 -1 bdev(0x55833b66cc00 td/osd-scrub-test/4/block) open stat got: (1) Operation not permitted 2026-02-13T02:28:58.768 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:58.762+0000 7fbd50310a40 -1 bluestore(td/osd-scrub-test/4) _read_fsid unparsable uuid 2026-02-13T02:28:59.436 INFO:tasks.workunit.client.0.trial112.stdout:adding osd4 key to auth repository 2026-02-13T02:28:59.648 INFO:tasks.workunit.client.0.trial112.stdout:start osd.4 2026-02-13T02:28:59.667 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:59.658+0000 7f0987d11a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:59.669 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:59.662+0000 7f0987d11a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:59.670 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:28:59.662+0000 7f0987d11a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:28:59.805 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:29:00.109 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:29:00.102+0000 7f0987d11a40 -1 Falling back to public interface 2026-02-13T02:29:00.920 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:29:00.910+0000 7f0987d11a40 -1 osd.4 0 log_to_monitors true 2026-02-13T02:29:00.973 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:29:02.139 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:29:02.301 INFO:tasks.workunit.client.0.trial112.stdout:osd.4 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/1291141566,v1:127.0.0.1:6835/1291141566] [v2:127.0.0.1:6836/1291141566,v1:127.0.0.1:6837/1291141566] exists,up 2d981b71-1b7b-42f4-b70b-4ef3cc129870 2026-02-13T02:29:02.506 INFO:tasks.workunit.client.0.trial112.stderr:pool 'test' created 2026-02-13T02:29:04.153 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836495 2026-02-13T02:29:04.324 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705675 2026-02-13T02:29:06.803 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574857 2026-02-13T02:29:06.977 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.3 seq 77309411334 2026-02-13T02:29:07.142 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.4 seq 98784247812 2026-02-13T02:29:09.032 INFO:tasks.workunit.client.0.trial112.stdout:standard_scrub_cluster: dbg: test pool is test 1 2026-02-13T02:29:09.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:757: TEST_abort_periodic_for_operator: local poolid=1 2026-02-13T02:29:09.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:758: TEST_abort_periodic_for_operator: local poolname=test 2026-02-13T02:29:09.033 INFO:tasks.workunit.client.0.trial112.stdout:Pool: test : 1 2026-02-13T02:29:09.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:759: TEST_abort_periodic_for_operator: echo 'Pool: test : 1' 2026-02-13T02:29:09.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:762: TEST_abort_periodic_for_operator: local saved_echo_flag=x 2026-02-13T02:29:09.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:763: TEST_abort_periodic_for_operator: set +x 2026-02-13T02:29:09.034 INFO:tasks.workunit.client.0.trial112.stderr:1+0 records in 2026-02-13T02:29:09.034 INFO:tasks.workunit.client.0.trial112.stderr:1+0 records out 2026-02-13T02:29:09.034 INFO:tasks.workunit.client.0.trial112.stderr:320 bytes copied, 3.4815e-05 s, 9.2 MB/s 2026-02-13T02:29:15.170 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:776: TEST_abort_periodic_for_operator: declare -A pg_pr 2026-02-13T02:29:15.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:777: TEST_abort_periodic_for_operator: declare -A pg_ac 2026-02-13T02:29:15.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:778: TEST_abort_periodic_for_operator: declare -A pg_po 2026-02-13T02:29:15.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:779: TEST_abort_periodic_for_operator: build_pg_dicts td/osd-scrub-test pg_pr pg_ac pg_po - 2026-02-13T02:29:15.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:318: build_pg_dicts: local dir=td/osd-scrub-test 2026-02-13T02:29:15.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:319: build_pg_dicts: local -n pg_primary_dict=pg_pr 2026-02-13T02:29:15.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:320: build_pg_dicts: local -n pg_acting_dict=pg_ac 2026-02-13T02:29:15.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:321: build_pg_dicts: local -n pg_pool_dict=pg_po 2026-02-13T02:29:15.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:322: build_pg_dicts: local infile=- 2026-02-13T02:29:15.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:324: build_pg_dicts: local extr_dbg=0 2026-02-13T02:29:15.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:327: build_pg_dicts: local saved_echo_flag=x 2026-02-13T02:29:15.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:328: build_pg_dicts: set +x 2026-02-13T02:29:15.315 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs_brief 2026-02-13T02:29:16.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:780: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-13T02:29:16.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-13T02:29:16.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-13T02:29:16.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-13T02:29:16.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-13T02:29:16.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-13T02:29:16.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-13T02:29:16.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-13T02:29:16.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-13T02:29:16.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-13T02:29:16.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-13T02:29:16.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-13T02:29:16.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-13T02:29:16.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-13T02:29:16.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-13T02:29:16.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-13T02:29:16.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-13T02:29:16.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-13T02:29:16.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-13T02:29:16.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-13T02:29:16.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-13T02:29:16.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-13T02:29:16.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-13T02:29:16.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-13T02:29:16.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-13T02:29:16.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-13T02:29:16.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-13T02:29:16.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-13T02:29:16.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-13T02:29:16.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-13T02:29:16.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-13T02:29:16.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-13T02:29:16.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-13T02:29:16.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:785: TEST_abort_periodic_for_operator: wait_initial_scrubs pg_pr 2026-02-13T02:29:16.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:691: wait_initial_scrubs: local -n pg_to_prim_dict=pg_pr 2026-02-13T02:29:16.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:692: wait_initial_scrubs: local extr_dbg=1 2026-02-13T02:29:16.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:696: wait_initial_scrubs: ceph tell 'osd.*' config set osd_scrub_min_interval 7200 2026-02-13T02:29:16.075 INFO:tasks.workunit.client.0.trial112.stdout:osd.0: { 2026-02-13T02:29:16.076 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_min_interval = '' " 2026-02-13T02:29:16.076 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:16.083 INFO:tasks.workunit.client.0.trial112.stdout:osd.1: { 2026-02-13T02:29:16.083 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_min_interval = '' " 2026-02-13T02:29:16.083 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:16.091 INFO:tasks.workunit.client.0.trial112.stdout:osd.2: { 2026-02-13T02:29:16.091 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_min_interval = '' " 2026-02-13T02:29:16.091 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:16.098 INFO:tasks.workunit.client.0.trial112.stdout:osd.3: { 2026-02-13T02:29:16.098 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_min_interval = '' " 2026-02-13T02:29:16.098 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:16.105 INFO:tasks.workunit.client.0.trial112.stdout:osd.4: { 2026-02-13T02:29:16.105 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_min_interval = '' " 2026-02-13T02:29:16.105 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:16.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:697: wait_initial_scrubs: ceph tell 'osd.*' config set osd_deep_scrub_interval 14400 2026-02-13T02:29:16.188 INFO:tasks.workunit.client.0.trial112.stdout:osd.0: { 2026-02-13T02:29:16.188 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_deep_scrub_interval = '' " 2026-02-13T02:29:16.189 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:16.196 INFO:tasks.workunit.client.0.trial112.stdout:osd.1: { 2026-02-13T02:29:16.196 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_deep_scrub_interval = '' " 2026-02-13T02:29:16.196 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:16.203 INFO:tasks.workunit.client.0.trial112.stdout:osd.2: { 2026-02-13T02:29:16.203 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_deep_scrub_interval = '' " 2026-02-13T02:29:16.203 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:16.211 INFO:tasks.workunit.client.0.trial112.stdout:osd.3: { 2026-02-13T02:29:16.211 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_deep_scrub_interval = '' " 2026-02-13T02:29:16.211 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:16.218 INFO:tasks.workunit.client.0.trial112.stdout:osd.4: { 2026-02-13T02:29:16.218 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_deep_scrub_interval = '' " 2026-02-13T02:29:16.218 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:16.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:698: wait_initial_scrubs: ceph tell 'osd.*' config set osd_max_scrubs 32 2026-02-13T02:29:16.302 INFO:tasks.workunit.client.0.trial112.stdout:osd.0: { 2026-02-13T02:29:16.302 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_max_scrubs = '' " 2026-02-13T02:29:16.303 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:16.310 INFO:tasks.workunit.client.0.trial112.stdout:osd.1: { 2026-02-13T02:29:16.310 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_max_scrubs = '' " 2026-02-13T02:29:16.310 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:16.317 INFO:tasks.workunit.client.0.trial112.stdout:osd.2: { 2026-02-13T02:29:16.317 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_max_scrubs = '' " 2026-02-13T02:29:16.317 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:16.324 INFO:tasks.workunit.client.0.trial112.stdout:osd.3: { 2026-02-13T02:29:16.324 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_max_scrubs = '' " 2026-02-13T02:29:16.324 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:16.331 INFO:tasks.workunit.client.0.trial112.stdout:osd.4: { 2026-02-13T02:29:16.332 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_max_scrubs = '' " 2026-02-13T02:29:16.332 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:16.339 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:699: wait_initial_scrubs: ceph tell 'osd.*' config set osd_scrub_sleep 1 2026-02-13T02:29:16.415 INFO:tasks.workunit.client.0.trial112.stdout:osd.0: { 2026-02-13T02:29:16.415 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_sleep = '' " 2026-02-13T02:29:16.415 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:16.422 INFO:tasks.workunit.client.0.trial112.stdout:osd.1: { 2026-02-13T02:29:16.422 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_sleep = '' " 2026-02-13T02:29:16.422 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:16.429 INFO:tasks.workunit.client.0.trial112.stdout:osd.2: { 2026-02-13T02:29:16.429 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_sleep = '' " 2026-02-13T02:29:16.429 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:16.436 INFO:tasks.workunit.client.0.trial112.stdout:osd.3: { 2026-02-13T02:29:16.436 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_sleep = '' " 2026-02-13T02:29:16.437 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:16.444 INFO:tasks.workunit.client.0.trial112.stdout:osd.4: { 2026-02-13T02:29:16.444 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_sleep = '' " 2026-02-13T02:29:16.444 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:16.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:700: wait_initial_scrubs: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_max 10 2026-02-13T02:29:16.515 INFO:tasks.workunit.client.0.trial112.stdout:osd.0: { 2026-02-13T02:29:16.515 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_shallow_scrub_chunk_max = '' " 2026-02-13T02:29:16.515 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:16.523 INFO:tasks.workunit.client.0.trial112.stdout:osd.1: { 2026-02-13T02:29:16.523 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_shallow_scrub_chunk_max = '' " 2026-02-13T02:29:16.523 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:16.530 INFO:tasks.workunit.client.0.trial112.stdout:osd.2: { 2026-02-13T02:29:16.530 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_shallow_scrub_chunk_max = '' " 2026-02-13T02:29:16.530 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:16.538 INFO:tasks.workunit.client.0.trial112.stdout:osd.3: { 2026-02-13T02:29:16.538 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_shallow_scrub_chunk_max = '' " 2026-02-13T02:29:16.538 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:16.545 INFO:tasks.workunit.client.0.trial112.stdout:osd.4: { 2026-02-13T02:29:16.545 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_shallow_scrub_chunk_max = '' " 2026-02-13T02:29:16.545 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:16.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:701: wait_initial_scrubs: ceph tell 'osd.*' config set osd_scrub_chunk_max 10 2026-02-13T02:29:16.618 INFO:tasks.workunit.client.0.trial112.stdout:osd.0: { 2026-02-13T02:29:16.618 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_chunk_max = '' " 2026-02-13T02:29:16.618 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:16.626 INFO:tasks.workunit.client.0.trial112.stdout:osd.1: { 2026-02-13T02:29:16.626 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_chunk_max = '' " 2026-02-13T02:29:16.626 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:16.633 INFO:tasks.workunit.client.0.trial112.stdout:osd.2: { 2026-02-13T02:29:16.633 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_chunk_max = '' " 2026-02-13T02:29:16.633 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:16.640 INFO:tasks.workunit.client.0.trial112.stdout:osd.3: { 2026-02-13T02:29:16.640 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_chunk_max = '' " 2026-02-13T02:29:16.640 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:16.647 INFO:tasks.workunit.client.0.trial112.stdout:osd.4: { 2026-02-13T02:29:16.648 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_chunk_max = '' " 2026-02-13T02:29:16.648 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:16.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-13T02:29:16.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-13T02:29:16.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.8' 2026-02-13T02:29:16.658 INFO:tasks.workunit.client.0.trial112.stdout:Scheduling initial scrub for 1.8 2026-02-13T02:29:16.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.8 scrub 2026-02-13T02:29:16.720 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:29:16.720 INFO:tasks.workunit.client.0.trial112.stdout: "deep": false, 2026-02-13T02:29:16.720 INFO:tasks.workunit.client.0.trial112.stdout: "must": true, 2026-02-13T02:29:16.721 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "0.000000" 2026-02-13T02:29:16.721 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:16.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-13T02:29:16.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-13T02:29:16.728 INFO:tasks.workunit.client.0.trial112.stdout:Scheduling initial scrub for 1.9 2026-02-13T02:29:16.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.9' 2026-02-13T02:29:16.729 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.9 scrub 2026-02-13T02:29:16.791 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:29:16.791 INFO:tasks.workunit.client.0.trial112.stdout: "deep": false, 2026-02-13T02:29:16.792 INFO:tasks.workunit.client.0.trial112.stdout: "must": true, 2026-02-13T02:29:16.792 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "0.000000" 2026-02-13T02:29:16.792 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:16.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-13T02:29:16.799 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-13T02:29:16.799 INFO:tasks.workunit.client.0.trial112.stdout:Scheduling initial scrub for 1.4 2026-02-13T02:29:16.799 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.4' 2026-02-13T02:29:16.799 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.4 scrub 2026-02-13T02:29:16.873 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:29:16.873 INFO:tasks.workunit.client.0.trial112.stdout: "deep": false, 2026-02-13T02:29:16.873 INFO:tasks.workunit.client.0.trial112.stdout: "must": true, 2026-02-13T02:29:16.873 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "0.000000" 2026-02-13T02:29:16.873 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:16.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-13T02:29:16.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-13T02:29:16.880 INFO:tasks.workunit.client.0.trial112.stdout:Scheduling initial scrub for 1.5 2026-02-13T02:29:16.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.5' 2026-02-13T02:29:16.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.5 scrub 2026-02-13T02:29:16.949 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:29:16.949 INFO:tasks.workunit.client.0.trial112.stdout: "deep": false, 2026-02-13T02:29:16.949 INFO:tasks.workunit.client.0.trial112.stdout: "must": true, 2026-02-13T02:29:16.949 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "0.000000" 2026-02-13T02:29:16.949 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:16.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-13T02:29:16.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-13T02:29:16.956 INFO:tasks.workunit.client.0.trial112.stdout:Scheduling initial scrub for 1.6 2026-02-13T02:29:16.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.6' 2026-02-13T02:29:16.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.6 scrub 2026-02-13T02:29:17.028 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:29:17.028 INFO:tasks.workunit.client.0.trial112.stdout: "deep": false, 2026-02-13T02:29:17.028 INFO:tasks.workunit.client.0.trial112.stdout: "must": true, 2026-02-13T02:29:17.028 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "0.000000" 2026-02-13T02:29:17.028 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:17.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-13T02:29:17.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-13T02:29:17.036 INFO:tasks.workunit.client.0.trial112.stdout:Scheduling initial scrub for 1.7 2026-02-13T02:29:17.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.7' 2026-02-13T02:29:17.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.7 scrub 2026-02-13T02:29:17.109 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:29:17.109 INFO:tasks.workunit.client.0.trial112.stdout: "deep": false, 2026-02-13T02:29:17.109 INFO:tasks.workunit.client.0.trial112.stdout: "must": true, 2026-02-13T02:29:17.109 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "0.000000" 2026-02-13T02:29:17.110 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:17.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-13T02:29:17.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-13T02:29:17.119 INFO:tasks.workunit.client.0.trial112.stdout:Scheduling initial scrub for 1.0 2026-02-13T02:29:17.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.0' 2026-02-13T02:29:17.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.0 scrub 2026-02-13T02:29:17.183 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:29:17.184 INFO:tasks.workunit.client.0.trial112.stdout: "deep": false, 2026-02-13T02:29:17.184 INFO:tasks.workunit.client.0.trial112.stdout: "must": true, 2026-02-13T02:29:17.184 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "0.000000" 2026-02-13T02:29:17.184 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:17.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-13T02:29:17.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-13T02:29:17.194 INFO:tasks.workunit.client.0.trial112.stdout:Scheduling initial scrub for 1.1 2026-02-13T02:29:17.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.1' 2026-02-13T02:29:17.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.1 scrub 2026-02-13T02:29:17.257 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:29:17.257 INFO:tasks.workunit.client.0.trial112.stdout: "deep": false, 2026-02-13T02:29:17.257 INFO:tasks.workunit.client.0.trial112.stdout: "must": true, 2026-02-13T02:29:17.257 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "0.000000" 2026-02-13T02:29:17.257 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:17.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-13T02:29:17.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-13T02:29:17.265 INFO:tasks.workunit.client.0.trial112.stdout:Scheduling initial scrub for 1.2 2026-02-13T02:29:17.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.2' 2026-02-13T02:29:17.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.2 scrub 2026-02-13T02:29:17.340 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:29:17.340 INFO:tasks.workunit.client.0.trial112.stdout: "deep": false, 2026-02-13T02:29:17.340 INFO:tasks.workunit.client.0.trial112.stdout: "must": true, 2026-02-13T02:29:17.341 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "0.000000" 2026-02-13T02:29:17.341 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:17.348 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-13T02:29:17.348 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-13T02:29:17.348 INFO:tasks.workunit.client.0.trial112.stdout:Scheduling initial scrub for 1.3 2026-02-13T02:29:17.348 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.3' 2026-02-13T02:29:17.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.3 scrub 2026-02-13T02:29:17.413 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:29:17.413 INFO:tasks.workunit.client.0.trial112.stdout: "deep": false, 2026-02-13T02:29:17.413 INFO:tasks.workunit.client.0.trial112.stdout: "must": true, 2026-02-13T02:29:17.413 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "0.000000" 2026-02-13T02:29:17.413 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:17.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-13T02:29:17.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-13T02:29:17.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.d' 2026-02-13T02:29:17.424 INFO:tasks.workunit.client.0.trial112.stdout:Scheduling initial scrub for 1.d 2026-02-13T02:29:17.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.d scrub 2026-02-13T02:29:17.488 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:29:17.488 INFO:tasks.workunit.client.0.trial112.stdout: "deep": false, 2026-02-13T02:29:17.488 INFO:tasks.workunit.client.0.trial112.stdout: "must": true, 2026-02-13T02:29:17.488 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "0.000000" 2026-02-13T02:29:17.488 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:17.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-13T02:29:17.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-13T02:29:17.496 INFO:tasks.workunit.client.0.trial112.stdout:Scheduling initial scrub for 1.e 2026-02-13T02:29:17.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.e' 2026-02-13T02:29:17.497 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.e scrub 2026-02-13T02:29:17.559 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:29:17.560 INFO:tasks.workunit.client.0.trial112.stdout: "deep": false, 2026-02-13T02:29:17.560 INFO:tasks.workunit.client.0.trial112.stdout: "must": true, 2026-02-13T02:29:17.560 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "0.000000" 2026-02-13T02:29:17.560 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:17.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-13T02:29:17.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-13T02:29:17.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.f' 2026-02-13T02:29:17.567 INFO:tasks.workunit.client.0.trial112.stdout:Scheduling initial scrub for 1.f 2026-02-13T02:29:17.568 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.f scrub 2026-02-13T02:29:17.643 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:29:17.644 INFO:tasks.workunit.client.0.trial112.stdout: "deep": false, 2026-02-13T02:29:17.644 INFO:tasks.workunit.client.0.trial112.stdout: "must": true, 2026-02-13T02:29:17.644 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "0.000000" 2026-02-13T02:29:17.644 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:17.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-13T02:29:17.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-13T02:29:17.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.a' 2026-02-13T02:29:17.654 INFO:tasks.workunit.client.0.trial112.stdout:Scheduling initial scrub for 1.a 2026-02-13T02:29:17.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.a scrub 2026-02-13T02:29:17.716 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:29:17.716 INFO:tasks.workunit.client.0.trial112.stdout: "deep": false, 2026-02-13T02:29:17.716 INFO:tasks.workunit.client.0.trial112.stdout: "must": true, 2026-02-13T02:29:17.716 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "0.000000" 2026-02-13T02:29:17.717 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:17.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-13T02:29:17.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-13T02:29:17.724 INFO:tasks.workunit.client.0.trial112.stdout:Scheduling initial scrub for 1.b 2026-02-13T02:29:17.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.b' 2026-02-13T02:29:17.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.b scrub 2026-02-13T02:29:17.798 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:29:17.798 INFO:tasks.workunit.client.0.trial112.stdout: "deep": false, 2026-02-13T02:29:17.798 INFO:tasks.workunit.client.0.trial112.stdout: "must": true, 2026-02-13T02:29:17.798 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "0.000000" 2026-02-13T02:29:17.798 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:17.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-13T02:29:17.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-13T02:29:17.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.c' 2026-02-13T02:29:17.806 INFO:tasks.workunit.client.0.trial112.stdout:Scheduling initial scrub for 1.c 2026-02-13T02:29:17.806 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.c scrub 2026-02-13T02:29:17.876 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:29:17.877 INFO:tasks.workunit.client.0.trial112.stdout: "deep": false, 2026-02-13T02:29:17.877 INFO:tasks.workunit.client.0.trial112.stdout: "must": true, 2026-02-13T02:29:17.877 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "0.000000" 2026-02-13T02:29:17.877 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:17.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:708: wait_initial_scrubs: sleep 1 2026-02-13T02:29:18.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:709: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-13T02:29:18.886 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:709: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-13T02:29:18.886 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:710: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-13T02:29:19.036 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:29:19.047 INFO:tasks.workunit.client.0.trial112.stdout:[ 2026-02-13T02:29:19.047 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:19.047 INFO:tasks.workunit.client.0.trial112.stdout: "pgid": "1.f", 2026-02-13T02:29:19.047 INFO:tasks.workunit.client.0.trial112.stdout: "last_scrub_duration": 0 2026-02-13T02:29:19.048 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:29:19.048 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:19.048 INFO:tasks.workunit.client.0.trial112.stdout: "pgid": "1.e", 2026-02-13T02:29:19.048 INFO:tasks.workunit.client.0.trial112.stdout: "last_scrub_duration": 0 2026-02-13T02:29:19.048 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:29:19.048 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:19.048 INFO:tasks.workunit.client.0.trial112.stdout: "pgid": "1.d", 2026-02-13T02:29:19.048 INFO:tasks.workunit.client.0.trial112.stdout: "last_scrub_duration": 0 2026-02-13T02:29:19.048 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:29:19.048 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:19.048 INFO:tasks.workunit.client.0.trial112.stdout: "pgid": "1.0", 2026-02-13T02:29:19.048 INFO:tasks.workunit.client.0.trial112.stdout: "last_scrub_duration": 0 2026-02-13T02:29:19.049 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:29:19.049 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:19.049 INFO:tasks.workunit.client.0.trial112.stdout: "pgid": "1.1", 2026-02-13T02:29:19.049 INFO:tasks.workunit.client.0.trial112.stdout: "last_scrub_duration": 0 2026-02-13T02:29:19.049 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:29:19.049 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:19.049 INFO:tasks.workunit.client.0.trial112.stdout: "pgid": "1.c", 2026-02-13T02:29:19.049 INFO:tasks.workunit.client.0.trial112.stdout: "last_scrub_duration": 0 2026-02-13T02:29:19.049 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:29:19.049 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:19.049 INFO:tasks.workunit.client.0.trial112.stdout: "pgid": "1.2", 2026-02-13T02:29:19.049 INFO:tasks.workunit.client.0.trial112.stdout: "last_scrub_duration": 0 2026-02-13T02:29:19.050 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:29:19.050 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:19.050 INFO:tasks.workunit.client.0.trial112.stdout: "pgid": "1.3", 2026-02-13T02:29:19.050 INFO:tasks.workunit.client.0.trial112.stdout: "last_scrub_duration": 0 2026-02-13T02:29:19.050 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:29:19.050 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:19.050 INFO:tasks.workunit.client.0.trial112.stdout: "pgid": "1.4", 2026-02-13T02:29:19.050 INFO:tasks.workunit.client.0.trial112.stdout: "last_scrub_duration": 0 2026-02-13T02:29:19.050 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:29:19.050 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:19.050 INFO:tasks.workunit.client.0.trial112.stdout: "pgid": "1.5", 2026-02-13T02:29:19.050 INFO:tasks.workunit.client.0.trial112.stdout: "last_scrub_duration": 0 2026-02-13T02:29:19.050 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:29:19.051 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:19.051 INFO:tasks.workunit.client.0.trial112.stdout: "pgid": "1.6", 2026-02-13T02:29:19.051 INFO:tasks.workunit.client.0.trial112.stdout: "last_scrub_duration": 0 2026-02-13T02:29:19.051 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:29:19.051 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:19.051 INFO:tasks.workunit.client.0.trial112.stdout: "pgid": "1.7", 2026-02-13T02:29:19.051 INFO:tasks.workunit.client.0.trial112.stdout: "last_scrub_duration": 0 2026-02-13T02:29:19.051 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:29:19.051 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:19.051 INFO:tasks.workunit.client.0.trial112.stdout: "pgid": "1.8", 2026-02-13T02:29:19.051 INFO:tasks.workunit.client.0.trial112.stdout: "last_scrub_duration": 0 2026-02-13T02:29:19.051 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:29:19.052 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:19.052 INFO:tasks.workunit.client.0.trial112.stdout: "pgid": "1.9", 2026-02-13T02:29:19.052 INFO:tasks.workunit.client.0.trial112.stdout: "last_scrub_duration": 0 2026-02-13T02:29:19.052 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:29:19.052 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:19.052 INFO:tasks.workunit.client.0.trial112.stdout: "pgid": "1.a", 2026-02-13T02:29:19.052 INFO:tasks.workunit.client.0.trial112.stdout: "last_scrub_duration": 0 2026-02-13T02:29:19.052 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:29:19.052 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:19.052 INFO:tasks.workunit.client.0.trial112.stdout: "pgid": "1.b", 2026-02-13T02:29:19.052 INFO:tasks.workunit.client.0.trial112.stdout: "last_scrub_duration": 0 2026-02-13T02:29:19.052 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:29:19.052 INFO:tasks.workunit.client.0.trial112.stdout:] 2026-02-13T02:29:19.053 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:712: wait_initial_scrubs: tout=20 2026-02-13T02:29:19.053 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:713: wait_initial_scrubs: '[' 20 -gt 0 ']' 2026-02-13T02:29:19.053 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:714: wait_initial_scrubs: sleep 1 2026-02-13T02:29:20.049 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:715: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-13T02:29:20.049 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:715: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-13T02:29:20.050 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:716: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-13T02:29:20.199 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:29:20.208 INFO:tasks.workunit.client.0.trial112.stdout:[ 2026-02-13T02:29:20.208 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:20.208 INFO:tasks.workunit.client.0.trial112.stdout: "pgid": "1.f", 2026-02-13T02:29:20.208 INFO:tasks.workunit.client.0.trial112.stdout: "last_scrub_duration": 0 2026-02-13T02:29:20.208 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:29:20.208 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:20.208 INFO:tasks.workunit.client.0.trial112.stdout: "pgid": "1.e", 2026-02-13T02:29:20.208 INFO:tasks.workunit.client.0.trial112.stdout: "last_scrub_duration": 0 2026-02-13T02:29:20.208 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:29:20.208 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:20.209 INFO:tasks.workunit.client.0.trial112.stdout: "pgid": "1.0", 2026-02-13T02:29:20.209 INFO:tasks.workunit.client.0.trial112.stdout: "last_scrub_duration": 0 2026-02-13T02:29:20.209 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:29:20.209 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:20.209 INFO:tasks.workunit.client.0.trial112.stdout: "pgid": "1.1", 2026-02-13T02:29:20.209 INFO:tasks.workunit.client.0.trial112.stdout: "last_scrub_duration": 0 2026-02-13T02:29:20.209 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:29:20.209 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:20.209 INFO:tasks.workunit.client.0.trial112.stdout: "pgid": "1.c", 2026-02-13T02:29:20.209 INFO:tasks.workunit.client.0.trial112.stdout: "last_scrub_duration": 0 2026-02-13T02:29:20.209 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:29:20.209 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:20.209 INFO:tasks.workunit.client.0.trial112.stdout: "pgid": "1.3", 2026-02-13T02:29:20.209 INFO:tasks.workunit.client.0.trial112.stdout: "last_scrub_duration": 0 2026-02-13T02:29:20.209 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:29:20.209 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:20.209 INFO:tasks.workunit.client.0.trial112.stdout: "pgid": "1.7", 2026-02-13T02:29:20.209 INFO:tasks.workunit.client.0.trial112.stdout: "last_scrub_duration": 0 2026-02-13T02:29:20.209 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:29:20.209 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:20.209 INFO:tasks.workunit.client.0.trial112.stdout: "pgid": "1.a", 2026-02-13T02:29:20.209 INFO:tasks.workunit.client.0.trial112.stdout: "last_scrub_duration": 0 2026-02-13T02:29:20.209 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:29:20.209 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:20.209 INFO:tasks.workunit.client.0.trial112.stdout: "pgid": "1.b", 2026-02-13T02:29:20.209 INFO:tasks.workunit.client.0.trial112.stdout: "last_scrub_duration": 0 2026-02-13T02:29:20.209 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:29:20.209 INFO:tasks.workunit.client.0.trial112.stdout:] 2026-02-13T02:29:20.209 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-13T02:29:20.209 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-13T02:29:20.209 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: wc -l 2026-02-13T02:29:20.363 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:29:20.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: not_done=38 2026-02-13T02:29:20.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:720: wait_initial_scrubs: '[' 38 -le 1 ']' 2026-02-13T02:29:20.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:723: wait_initial_scrubs: not_done=9 2026-02-13T02:29:20.373 INFO:tasks.workunit.client.0.trial112.stdout:Still waiting for 9 PGs to finish initial scrubs (timeout 20) 2026-02-13T02:29:20.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:724: wait_initial_scrubs: echo 'Still waiting for 9 PGs to finish initial scrubs (timeout 20)' 2026-02-13T02:29:20.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:725: wait_initial_scrubs: tout=19 2026-02-13T02:29:20.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:713: wait_initial_scrubs: '[' 19 -gt 0 ']' 2026-02-13T02:29:20.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:714: wait_initial_scrubs: sleep 1 2026-02-13T02:29:21.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:715: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-13T02:29:21.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:715: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-13T02:29:21.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:716: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-13T02:29:21.523 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:29:21.532 INFO:tasks.workunit.client.0.trial112.stdout:[ 2026-02-13T02:29:21.533 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:21.533 INFO:tasks.workunit.client.0.trial112.stdout: "pgid": "1.f", 2026-02-13T02:29:21.533 INFO:tasks.workunit.client.0.trial112.stdout: "last_scrub_duration": 0 2026-02-13T02:29:21.533 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:29:21.533 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:21.533 INFO:tasks.workunit.client.0.trial112.stdout: "pgid": "1.e", 2026-02-13T02:29:21.533 INFO:tasks.workunit.client.0.trial112.stdout: "last_scrub_duration": 0 2026-02-13T02:29:21.533 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:29:21.533 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:21.533 INFO:tasks.workunit.client.0.trial112.stdout: "pgid": "1.0", 2026-02-13T02:29:21.534 INFO:tasks.workunit.client.0.trial112.stdout: "last_scrub_duration": 0 2026-02-13T02:29:21.534 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:29:21.534 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:21.534 INFO:tasks.workunit.client.0.trial112.stdout: "pgid": "1.1", 2026-02-13T02:29:21.534 INFO:tasks.workunit.client.0.trial112.stdout: "last_scrub_duration": 0 2026-02-13T02:29:21.534 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:29:21.534 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:21.534 INFO:tasks.workunit.client.0.trial112.stdout: "pgid": "1.c", 2026-02-13T02:29:21.534 INFO:tasks.workunit.client.0.trial112.stdout: "last_scrub_duration": 0 2026-02-13T02:29:21.534 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:29:21.534 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:21.534 INFO:tasks.workunit.client.0.trial112.stdout: "pgid": "1.3", 2026-02-13T02:29:21.534 INFO:tasks.workunit.client.0.trial112.stdout: "last_scrub_duration": 0 2026-02-13T02:29:21.535 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:29:21.535 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:21.535 INFO:tasks.workunit.client.0.trial112.stdout: "pgid": "1.7", 2026-02-13T02:29:21.535 INFO:tasks.workunit.client.0.trial112.stdout: "last_scrub_duration": 0 2026-02-13T02:29:21.535 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:29:21.535 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:21.535 INFO:tasks.workunit.client.0.trial112.stdout: "pgid": "1.a", 2026-02-13T02:29:21.535 INFO:tasks.workunit.client.0.trial112.stdout: "last_scrub_duration": 0 2026-02-13T02:29:21.535 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:29:21.535 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:21.535 INFO:tasks.workunit.client.0.trial112.stdout: "pgid": "1.b", 2026-02-13T02:29:21.535 INFO:tasks.workunit.client.0.trial112.stdout: "last_scrub_duration": 0 2026-02-13T02:29:21.536 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:29:21.536 INFO:tasks.workunit.client.0.trial112.stdout:] 2026-02-13T02:29:21.536 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-13T02:29:21.536 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-13T02:29:21.537 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: wc -l 2026-02-13T02:29:21.682 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:29:21.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: not_done=38 2026-02-13T02:29:21.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:720: wait_initial_scrubs: '[' 38 -le 1 ']' 2026-02-13T02:29:21.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:723: wait_initial_scrubs: not_done=9 2026-02-13T02:29:21.694 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:724: wait_initial_scrubs: echo 'Still waiting for 9 PGs to finish initial scrubs (timeout 19)' 2026-02-13T02:29:21.694 INFO:tasks.workunit.client.0.trial112.stdout:Still waiting for 9 PGs to finish initial scrubs (timeout 19) 2026-02-13T02:29:21.694 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:725: wait_initial_scrubs: tout=18 2026-02-13T02:29:21.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:713: wait_initial_scrubs: '[' 18 -gt 0 ']' 2026-02-13T02:29:21.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:714: wait_initial_scrubs: sleep 1 2026-02-13T02:29:22.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:715: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-13T02:29:22.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:715: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-13T02:29:22.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:716: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-13T02:29:22.845 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:29:22.854 INFO:tasks.workunit.client.0.trial112.stdout:[ 2026-02-13T02:29:22.854 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:22.854 INFO:tasks.workunit.client.0.trial112.stdout: "pgid": "1.e", 2026-02-13T02:29:22.854 INFO:tasks.workunit.client.0.trial112.stdout: "last_scrub_duration": 0 2026-02-13T02:29:22.854 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:29:22.854 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:22.855 INFO:tasks.workunit.client.0.trial112.stdout: "pgid": "1.a", 2026-02-13T02:29:22.855 INFO:tasks.workunit.client.0.trial112.stdout: "last_scrub_duration": 0 2026-02-13T02:29:22.855 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:29:22.855 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:22.855 INFO:tasks.workunit.client.0.trial112.stdout: "pgid": "1.b", 2026-02-13T02:29:22.855 INFO:tasks.workunit.client.0.trial112.stdout: "last_scrub_duration": 0 2026-02-13T02:29:22.855 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:29:22.855 INFO:tasks.workunit.client.0.trial112.stdout:] 2026-02-13T02:29:22.856 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-13T02:29:22.856 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-13T02:29:22.856 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: wc -l 2026-02-13T02:29:23.001 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:29:23.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: not_done=14 2026-02-13T02:29:23.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:720: wait_initial_scrubs: '[' 14 -le 1 ']' 2026-02-13T02:29:23.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:723: wait_initial_scrubs: not_done=3 2026-02-13T02:29:23.012 INFO:tasks.workunit.client.0.trial112.stdout:Still waiting for 3 PGs to finish initial scrubs (timeout 18) 2026-02-13T02:29:23.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:724: wait_initial_scrubs: echo 'Still waiting for 3 PGs to finish initial scrubs (timeout 18)' 2026-02-13T02:29:23.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:725: wait_initial_scrubs: tout=17 2026-02-13T02:29:23.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:713: wait_initial_scrubs: '[' 17 -gt 0 ']' 2026-02-13T02:29:23.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:714: wait_initial_scrubs: sleep 1 2026-02-13T02:29:24.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:715: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-13T02:29:24.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:715: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-13T02:29:24.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:716: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-13T02:29:24.163 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:29:24.172 INFO:tasks.workunit.client.0.trial112.stdout:[ 2026-02-13T02:29:24.172 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:24.172 INFO:tasks.workunit.client.0.trial112.stdout: "pgid": "1.b", 2026-02-13T02:29:24.172 INFO:tasks.workunit.client.0.trial112.stdout: "last_scrub_duration": 0 2026-02-13T02:29:24.173 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:29:24.173 INFO:tasks.workunit.client.0.trial112.stdout:] 2026-02-13T02:29:24.174 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-13T02:29:24.174 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-13T02:29:24.174 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: wc -l 2026-02-13T02:29:24.324 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:29:24.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: not_done=6 2026-02-13T02:29:24.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:720: wait_initial_scrubs: '[' 6 -le 1 ']' 2026-02-13T02:29:24.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:723: wait_initial_scrubs: not_done=1 2026-02-13T02:29:24.335 INFO:tasks.workunit.client.0.trial112.stdout:Still waiting for 1 PGs to finish initial scrubs (timeout 17) 2026-02-13T02:29:24.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:724: wait_initial_scrubs: echo 'Still waiting for 1 PGs to finish initial scrubs (timeout 17)' 2026-02-13T02:29:24.336 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:725: wait_initial_scrubs: tout=16 2026-02-13T02:29:24.336 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:713: wait_initial_scrubs: '[' 16 -gt 0 ']' 2026-02-13T02:29:24.336 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:714: wait_initial_scrubs: sleep 1 2026-02-13T02:29:25.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:715: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-13T02:29:25.336 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:715: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-13T02:29:25.336 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:716: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-13T02:29:25.484 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:29:25.493 INFO:tasks.workunit.client.0.trial112.stdout:[ 2026-02-13T02:29:25.493 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:25.493 INFO:tasks.workunit.client.0.trial112.stdout: "pgid": "1.b", 2026-02-13T02:29:25.493 INFO:tasks.workunit.client.0.trial112.stdout: "last_scrub_duration": 0 2026-02-13T02:29:25.493 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:29:25.493 INFO:tasks.workunit.client.0.trial112.stdout:] 2026-02-13T02:29:25.494 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-13T02:29:25.494 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-13T02:29:25.494 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: wc -l 2026-02-13T02:29:25.642 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:29:25.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: not_done=6 2026-02-13T02:29:25.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:720: wait_initial_scrubs: '[' 6 -le 1 ']' 2026-02-13T02:29:25.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:723: wait_initial_scrubs: not_done=1 2026-02-13T02:29:25.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:724: wait_initial_scrubs: echo 'Still waiting for 1 PGs to finish initial scrubs (timeout 16)' 2026-02-13T02:29:25.653 INFO:tasks.workunit.client.0.trial112.stdout:Still waiting for 1 PGs to finish initial scrubs (timeout 16) 2026-02-13T02:29:25.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:725: wait_initial_scrubs: tout=15 2026-02-13T02:29:25.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:713: wait_initial_scrubs: '[' 15 -gt 0 ']' 2026-02-13T02:29:25.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:714: wait_initial_scrubs: sleep 1 2026-02-13T02:29:26.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:715: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-13T02:29:26.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:715: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-13T02:29:26.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:716: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-13T02:29:26.802 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:29:26.812 INFO:tasks.workunit.client.0.trial112.stdout:[] 2026-02-13T02:29:26.813 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-13T02:29:26.813 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-13T02:29:26.813 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: wc -l 2026-02-13T02:29:26.961 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:29:26.970 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: not_done=1 2026-02-13T02:29:26.970 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:720: wait_initial_scrubs: '[' 1 -le 1 ']' 2026-02-13T02:29:26.970 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:721: wait_initial_scrubs: break 2026-02-13T02:29:26.970 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:727: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-13T02:29:26.971 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:728: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-13T02:29:27.123 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:29:27.133 INFO:tasks.workunit.client.0.trial112.stdout:[] 2026-02-13T02:29:27.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:729: wait_initial_scrubs: (( tout == 0 )) 2026-02-13T02:29:27.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:730: wait_initial_scrubs: return 0 2026-02-13T02:29:27.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:788: TEST_abort_periodic_for_operator: ceph tell 'osd.*' config set osd_max_scrubs 1 2026-02-13T02:29:27.195 INFO:tasks.workunit.client.0.trial112.stdout:osd.0: { 2026-02-13T02:29:27.195 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_max_scrubs = '' " 2026-02-13T02:29:27.196 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:27.203 INFO:tasks.workunit.client.0.trial112.stdout:osd.1: { 2026-02-13T02:29:27.203 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_max_scrubs = '' " 2026-02-13T02:29:27.203 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:27.211 INFO:tasks.workunit.client.0.trial112.stdout:osd.2: { 2026-02-13T02:29:27.211 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_max_scrubs = '' " 2026-02-13T02:29:27.211 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:27.218 INFO:tasks.workunit.client.0.trial112.stdout:osd.3: { 2026-02-13T02:29:27.218 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_max_scrubs = '' " 2026-02-13T02:29:27.218 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:27.225 INFO:tasks.workunit.client.0.trial112.stdout:osd.4: { 2026-02-13T02:29:27.225 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_max_scrubs = '' " 2026-02-13T02:29:27.226 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:27.233 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:789: TEST_abort_periodic_for_operator: ceph tell 'osd.*' config set osd_stats_update_period_not_scrubbing 1 2026-02-13T02:29:27.308 INFO:tasks.workunit.client.0.trial112.stdout:osd.0: { 2026-02-13T02:29:27.308 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_stats_update_period_not_scrubbing = '' " 2026-02-13T02:29:27.309 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:27.316 INFO:tasks.workunit.client.0.trial112.stdout:osd.1: { 2026-02-13T02:29:27.316 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_stats_update_period_not_scrubbing = '' " 2026-02-13T02:29:27.316 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:27.323 INFO:tasks.workunit.client.0.trial112.stdout:osd.2: { 2026-02-13T02:29:27.323 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_stats_update_period_not_scrubbing = '' " 2026-02-13T02:29:27.323 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:27.331 INFO:tasks.workunit.client.0.trial112.stdout:osd.3: { 2026-02-13T02:29:27.331 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_stats_update_period_not_scrubbing = '' " 2026-02-13T02:29:27.331 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:27.338 INFO:tasks.workunit.client.0.trial112.stdout:osd.4: { 2026-02-13T02:29:27.338 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_stats_update_period_not_scrubbing = '' " 2026-02-13T02:29:27.338 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:27.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:792: TEST_abort_periodic_for_operator: ceph tell 'osd.*' config set osd_scrub_sleep 3 2026-02-13T02:29:27.420 INFO:tasks.workunit.client.0.trial112.stdout:osd.0: { 2026-02-13T02:29:27.420 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_sleep = '' " 2026-02-13T02:29:27.420 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:27.427 INFO:tasks.workunit.client.0.trial112.stdout:osd.1: { 2026-02-13T02:29:27.427 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_sleep = '' " 2026-02-13T02:29:27.427 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:27.434 INFO:tasks.workunit.client.0.trial112.stdout:osd.2: { 2026-02-13T02:29:27.434 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_sleep = '' " 2026-02-13T02:29:27.434 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:27.441 INFO:tasks.workunit.client.0.trial112.stdout:osd.3: { 2026-02-13T02:29:27.441 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_sleep = '' " 2026-02-13T02:29:27.442 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:27.449 INFO:tasks.workunit.client.0.trial112.stdout:osd.4: { 2026-02-13T02:29:27.449 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_sleep = '' " 2026-02-13T02:29:27.449 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:27.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:793: TEST_abort_periodic_for_operator: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_max 2 2026-02-13T02:29:27.523 INFO:tasks.workunit.client.0.trial112.stdout:osd.0: { 2026-02-13T02:29:27.523 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_shallow_scrub_chunk_max = '' " 2026-02-13T02:29:27.523 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:27.530 INFO:tasks.workunit.client.0.trial112.stdout:osd.1: { 2026-02-13T02:29:27.530 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_shallow_scrub_chunk_max = '' " 2026-02-13T02:29:27.530 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:27.537 INFO:tasks.workunit.client.0.trial112.stdout:osd.2: { 2026-02-13T02:29:27.538 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_shallow_scrub_chunk_max = '' " 2026-02-13T02:29:27.538 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:27.544 INFO:tasks.workunit.client.0.trial112.stdout:osd.3: { 2026-02-13T02:29:27.545 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_shallow_scrub_chunk_max = '' " 2026-02-13T02:29:27.545 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:27.552 INFO:tasks.workunit.client.0.trial112.stdout:osd.4: { 2026-02-13T02:29:27.552 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_shallow_scrub_chunk_max = '' " 2026-02-13T02:29:27.552 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:27.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:794: TEST_abort_periodic_for_operator: ceph tell 'osd.*' config set osd_scrub_chunk_max 2 2026-02-13T02:29:27.626 INFO:tasks.workunit.client.0.trial112.stdout:osd.0: { 2026-02-13T02:29:27.626 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_chunk_max = '' " 2026-02-13T02:29:27.626 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:27.634 INFO:tasks.workunit.client.0.trial112.stdout:osd.1: { 2026-02-13T02:29:27.634 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_chunk_max = '' " 2026-02-13T02:29:27.634 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:27.641 INFO:tasks.workunit.client.0.trial112.stdout:osd.2: { 2026-02-13T02:29:27.641 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_chunk_max = '' " 2026-02-13T02:29:27.641 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:27.649 INFO:tasks.workunit.client.0.trial112.stdout:osd.3: { 2026-02-13T02:29:27.649 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_chunk_max = '' " 2026-02-13T02:29:27.649 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:27.656 INFO:tasks.workunit.client.0.trial112.stdout:osd.4: { 2026-02-13T02:29:27.656 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_chunk_max = '' " 2026-02-13T02:29:27.656 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:27.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:795: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-13T02:29:27.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:798: TEST_abort_periodic_for_operator: local pg2= 2026-02-13T02:29:27.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:799: TEST_abort_periodic_for_operator: for pg1 in "${!pg_pr[@]}" 2026-02-13T02:29:27.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:800: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-13T02:29:27.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:801: TEST_abort_periodic_for_operator: [[ 1.8 == \1\.\8 ]] 2026-02-13T02:29:27.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:802: TEST_abort_periodic_for_operator: continue 2026-02-13T02:29:27.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:800: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-13T02:29:27.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:801: TEST_abort_periodic_for_operator: [[ 1.9 == \1\.\8 ]] 2026-02-13T02:29:27.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:804: TEST_abort_periodic_for_operator: [[ 1 == \1 ]] 2026-02-13T02:29:27.667 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:805: TEST_abort_periodic_for_operator: count_common_active 1.9 1.8 pg_ac 2026-02-13T02:29:27.667 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:367: count_common_active: local pg1=1.9 2026-02-13T02:29:27.667 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:368: count_common_active: local pg2=1.8 2026-02-13T02:29:27.667 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:369: count_common_active: local -n pg_acting_dict=pg_ac 2026-02-13T02:29:27.667 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:371: count_common_active: a1=('1' '4' '2') 2026-02-13T02:29:27.667 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:371: count_common_active: local -a a1 2026-02-13T02:29:27.668 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:372: count_common_active: a2=('1' '4' '3') 2026-02-13T02:29:27.668 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:372: count_common_active: local -a a2 2026-02-13T02:29:27.668 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:374: count_common_active: local -i cnt=0 2026-02-13T02:29:27.668 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:375: count_common_active: for i in "${a1[@]}" 2026-02-13T02:29:27.668 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:376: count_common_active: for j in "${a2[@]}" 2026-02-13T02:29:27.668 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:377: count_common_active: [[ 1 -eq 1 ]] 2026-02-13T02:29:27.668 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:378: count_common_active: cnt=1 2026-02-13T02:29:27.668 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:376: count_common_active: for j in "${a2[@]}" 2026-02-13T02:29:27.668 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:377: count_common_active: [[ 1 -eq 4 ]] 2026-02-13T02:29:27.668 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:376: count_common_active: for j in "${a2[@]}" 2026-02-13T02:29:27.668 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:377: count_common_active: [[ 1 -eq 3 ]] 2026-02-13T02:29:27.668 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:375: count_common_active: for i in "${a1[@]}" 2026-02-13T02:29:27.668 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:376: count_common_active: for j in "${a2[@]}" 2026-02-13T02:29:27.669 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:377: count_common_active: [[ 4 -eq 1 ]] 2026-02-13T02:29:27.669 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:376: count_common_active: for j in "${a2[@]}" 2026-02-13T02:29:27.669 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:377: count_common_active: [[ 4 -eq 4 ]] 2026-02-13T02:29:27.669 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:378: count_common_active: cnt=2 2026-02-13T02:29:27.669 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:376: count_common_active: for j in "${a2[@]}" 2026-02-13T02:29:27.669 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:377: count_common_active: [[ 4 -eq 3 ]] 2026-02-13T02:29:27.669 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:375: count_common_active: for i in "${a1[@]}" 2026-02-13T02:29:27.669 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:376: count_common_active: for j in "${a2[@]}" 2026-02-13T02:29:27.669 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:377: count_common_active: [[ 2 -eq 1 ]] 2026-02-13T02:29:27.669 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:376: count_common_active: for j in "${a2[@]}" 2026-02-13T02:29:27.669 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:377: count_common_active: [[ 2 -eq 4 ]] 2026-02-13T02:29:27.669 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:376: count_common_active: for j in "${a2[@]}" 2026-02-13T02:29:27.669 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:377: count_common_active: [[ 2 -eq 3 ]] 2026-02-13T02:29:27.670 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:383: count_common_active: printf %d 2 2026-02-13T02:29:27.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:805: TEST_abort_periodic_for_operator: local -i common=2 2026-02-13T02:29:27.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:806: TEST_abort_periodic_for_operator: [[ 2 -gt 1 ]] 2026-02-13T02:29:27.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:807: TEST_abort_periodic_for_operator: pg2=1.9 2026-02-13T02:29:27.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:808: TEST_abort_periodic_for_operator: break 2 2026-02-13T02:29:27.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:814: TEST_abort_periodic_for_operator: [[ -z 1.9 ]] 2026-02-13T02:29:27.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:820: TEST_abort_periodic_for_operator: ceph tell osd.1 config set osd_max_scrubs 2 2026-02-13T02:29:27.732 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:29:27.732 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_max_scrubs = '' " 2026-02-13T02:29:27.732 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:27.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:821: TEST_abort_periodic_for_operator: echo 'The two PGs to manipulate are 1.8 and 1.9' 2026-02-13T02:29:27.742 INFO:tasks.workunit.client.0.trial112.stdout:The two PGs to manipulate are 1.8 and 1.9 2026-02-13T02:29:27.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:823: TEST_abort_periodic_for_operator: set_query_debug 1.8 2026-02-13T02:29:27.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:417: set_query_debug: local pgid=1.8 2026-02-13T02:29:27.743 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:419: set_query_debug: ceph pg dump pgs_brief 2026-02-13T02:29:27.743 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:419: set_query_debug: awk -v 'pg=^1.8' -n -e '$0 ~ pg { print(gensub(/[^0-9]*([0-9]+).*/,"\\1","g",$5)); }' 2026-02-13T02:29:27.894 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs_brief 2026-02-13T02:29:27.905 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:419: set_query_debug: local prim_osd=1 2026-02-13T02:29:27.905 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:421: set_query_debug: echo 'Setting scrub debug data. Primary for 1.8 is 1' 2026-02-13T02:29:27.905 INFO:tasks.workunit.client.0.trial112.stdout:Setting scrub debug data. Primary for 1.8 is 1 2026-02-13T02:29:27.907 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:422: set_query_debug: get_asok_path osd.1 2026-02-13T02:29:27.907 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-13T02:29:27.907 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-13T02:29:27.907 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:29:27.907 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:29:27.907 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:29:27.907 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.508946/ceph-osd.1.asok 2026-02-13T02:29:27.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:422: set_query_debug: CEPH_ARGS= 2026-02-13T02:29:27.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:422: set_query_debug: ceph --format=json daemon /tmp/ceph-asok.508946/ceph-osd.1.asok scrubdebug 1.8 set sessions 2026-02-13T02:29:27.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:826: TEST_abort_periodic_for_operator: local is_act 2026-02-13T02:29:27.976 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:827: TEST_abort_periodic_for_operator: seq 1 3 2026-02-13T02:29:27.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:827: TEST_abort_periodic_for_operator: for i in $( seq 1 3 ) 2026-02-13T02:29:27.978 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:829: TEST_abort_periodic_for_operator: ceph pg 1.8 query 2026-02-13T02:29:27.978 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:829: TEST_abort_periodic_for_operator: jq .scrubber.active 2026-02-13T02:29:28.048 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:829: TEST_abort_periodic_for_operator: is_act=false 2026-02-13T02:29:28.048 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:830: TEST_abort_periodic_for_operator: [[ false = \f\a\l\s\e ]] 2026-02-13T02:29:28.048 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:831: TEST_abort_periodic_for_operator: break 2026-02-13T02:29:28.048 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:836: TEST_abort_periodic_for_operator: ceph pg dump pgs 2026-02-13T02:29:28.195 INFO:tasks.workunit.client.0.trial112.stdout:{"success":true}PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-13T02:29:28.195 INFO:tasks.workunit.client.0.trial112.stdout:1.f 19 0 0 0 0 6080 0 0 19 0 19 active+clean 2026-02-13T02:29:20.243184+0000 27'19 27:44 [3,0,4] 3 [3,0,4] 3 27'19 2026-02-13T02:29:20.243123+0000 0'0 2026-02-13T02:29:02.500747+0000 0 2 periodic scrub scheduled @ 2026-02-13T04:29:20.243123+0000 19 0 2026-02-13T02:29:28.195 INFO:tasks.workunit.client.0.trial112.stdout:1.e 20 0 0 0 0 6400 0 0 20 0 20 active+clean 2026-02-13T02:29:22.064316+0000 27'20 27:46 [1,4,2] 1 [1,4,2] 1 27'20 2026-02-13T02:29:22.064247+0000 0'0 2026-02-13T02:29:02.500747+0000 0 2 periodic scrub scheduled @ 2026-02-13T04:29:22.064247+0000 20 0 2026-02-13T02:29:28.196 INFO:tasks.workunit.client.0.trial112.stdout:1.d 14 0 0 0 0 4480 0 0 14 0 14 active+clean 2026-02-13T02:29:18.588445+0000 27'14 27:39 [2,4,1] 2 [2,4,1] 2 27'14 2026-02-13T02:29:18.588381+0000 0'0 2026-02-13T02:29:02.500747+0000 0 2 periodic scrub scheduled @ 2026-02-13T04:29:18.588381+0000 14 0 2026-02-13T02:29:28.196 INFO:tasks.workunit.client.0.trial112.stdout:1.0 22 0 0 0 0 7040 0 0 22 0 22 active+clean 2026-02-13T02:29:21.030696+0000 27'22 27:53 [1,0,2] 1 [1,0,2] 1 27'22 2026-02-13T02:29:21.030644+0000 0'0 2026-02-13T02:29:02.500747+0000 0 1 periodic scrub scheduled @ 2026-02-13T04:29:21.030644+0000 22 0 2026-02-13T02:29:28.196 INFO:tasks.workunit.client.0.trial112.stdout:1.1 12 0 0 0 0 3840 0 0 12 0 12 active+clean 2026-02-13T02:29:19.745970+0000 27'12 27:41 [4,3,0] 4 [4,3,0] 4 27'12 2026-02-13T02:29:19.745907+0000 0'0 2026-02-13T02:29:02.500747+0000 0 1 periodic scrub scheduled @ 2026-02-13T04:29:19.745907+0000 12 0 2026-02-13T02:29:28.196 INFO:tasks.workunit.client.0.trial112.stdout:1.c 15 0 0 0 0 4800 0 0 15 0 15 active+clean 2026-02-13T02:29:21.244701+0000 27'15 27:41 [3,0,4] 3 [3,0,4] 3 27'15 2026-02-13T02:29:21.244628+0000 0'0 2026-02-13T02:29:02.500747+0000 0 2 periodic scrub scheduled @ 2026-02-13T04:29:21.244628+0000 15 0 2026-02-13T02:29:28.196 INFO:tasks.workunit.client.0.trial112.stdout:1.2 15 0 0 0 0 4800 0 0 15 0 15 active+clean 2026-02-13T02:29:18.578715+0000 27'15 27:43 [0,4,1] 0 [0,4,1] 0 27'15 2026-02-13T02:29:18.578632+0000 0'0 2026-02-13T02:29:02.500747+0000 0 1 periodic scrub scheduled @ 2026-02-13T04:29:18.578632+0000 15 0 2026-02-13T02:29:28.196 INFO:tasks.workunit.client.0.trial112.stdout:1.3 19 0 0 0 0 6080 0 0 19 0 19 active+clean 2026-02-13T02:29:20.725804+0000 27'19 27:47 [4,1,2] 4 [4,1,2] 4 27'19 2026-02-13T02:29:20.725747+0000 0'0 2026-02-13T02:29:02.500747+0000 0 1 periodic scrub scheduled @ 2026-02-13T04:29:20.725747+0000 19 0 2026-02-13T02:29:28.196 INFO:tasks.workunit.client.0.trial112.stdout:1.4 15 0 0 0 0 4800 0 0 15 0 15 active+clean 2026-02-13T02:29:18.227858+0000 27'15 27:42 [3,0,1] 3 [3,0,1] 3 27'15 2026-02-13T02:29:18.227813+0000 0'0 2026-02-13T02:29:02.500747+0000 0 2 periodic scrub scheduled @ 2026-02-13T04:29:18.227813+0000 15 0 2026-02-13T02:29:28.196 INFO:tasks.workunit.client.0.trial112.stdout:1.5 26 0 0 0 0 8320 0 0 26 0 26 active+clean 2026-02-13T02:29:18.784234+0000 27'26 27:53 [4,2,3] 4 [4,2,3] 4 27'26 2026-02-13T02:29:18.784175+0000 0'0 2026-02-13T02:29:02.500747+0000 0 2 periodic scrub scheduled @ 2026-02-13T04:29:18.784175+0000 26 0 2026-02-13T02:29:28.196 INFO:tasks.workunit.client.0.trial112.stdout:1.6 12 0 0 0 0 3840 0 0 12 0 12 active+clean 2026-02-13T02:29:19.236520+0000 27'12 27:39 [3,4,2] 3 [3,4,2] 3 27'12 2026-02-13T02:29:19.236445+0000 0'0 2026-02-13T02:29:02.500747+0000 0 2 periodic scrub scheduled @ 2026-02-13T04:29:19.236445+0000 12 0 2026-02-13T02:29:28.196 INFO:tasks.workunit.client.0.trial112.stdout:1.7 11 0 0 0 0 3520 0 0 11 0 11 active+clean 2026-02-13T02:29:20.026398+0000 27'11 27:37 [1,3,0] 1 [1,3,0] 1 27'11 2026-02-13T02:29:20.026350+0000 0'0 2026-02-13T02:29:02.500747+0000 0 1 periodic scrub scheduled @ 2026-02-13T04:29:20.026350+0000 11 0 2026-02-13T02:29:28.197 INFO:tasks.workunit.client.0.trial112.stdout:1.8 28 0 0 0 0 8960 0 0 28 0 28 active+clean 2026-02-13T02:29:17.971388+0000 27'28 27:54 [1,4,3] 1 [1,4,3] 1 27'28 2026-02-13T02:29:17.971316+0000 0'0 2026-02-13T02:29:02.500747+0000 0 2 periodic scrub scheduled @ 2026-02-13T04:29:17.971316+0000 28 0 2026-02-13T02:29:28.197 INFO:tasks.workunit.client.0.trial112.stdout:1.9 8 0 0 0 0 2560 0 0 8 0 8 active+clean 2026-02-13T02:29:18.997847+0000 27'8 27:34 [1,4,2] 1 [1,4,2] 1 27'8 2026-02-13T02:29:18.997801+0000 0'0 2026-02-13T02:29:02.500747+0000 0 1 periodic scrub scheduled @ 2026-02-13T04:29:18.997801+0000 8 0 2026-02-13T02:29:28.197 INFO:tasks.workunit.client.0.trial112.stdout:1.a 9 0 0 0 0 2880 0 0 9 0 9 active+clean 2026-02-13T02:29:23.099672+0000 27'9 27:35 [1,2,4] 1 [1,2,4] 1 27'9 2026-02-13T02:29:23.099617+0000 0'0 2026-02-13T02:29:02.500747+0000 0 2 periodic scrub scheduled @ 2026-02-13T04:29:23.099617+0000 9 0 2026-02-13T02:29:28.197 INFO:tasks.workunit.client.0.trial112.stdout:1.b 11 0 0 0 0 3520 0 0 11 0 11 active+clean 2026-02-13T02:29:24.089223+0000 27'11 27:37 [1,0,2] 1 [1,0,2] 1 27'11 2026-02-13T02:29:24.089168+0000 0'0 2026-02-13T02:29:02.500747+0000 0 1 periodic scrub scheduled @ 2026-02-13T04:29:24.089168+0000 11 0 2026-02-13T02:29:28.197 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T02:29:28.197 INFO:tasks.workunit.client.0.trial112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-13T02:29:28.197 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:29:28.206 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:837: TEST_abort_periodic_for_operator: [[ false != \f\a\l\s\e ]] 2026-02-13T02:29:28.206 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:842: TEST_abort_periodic_for_operator: sleep 0.5 2026-02-13T02:29:28.707 INFO:tasks.workunit.client.0.trial112.stdout:Initiating a periodic scrub of 1.8 2026-02-13T02:29:28.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:844: TEST_abort_periodic_for_operator: echo 'Initiating a periodic scrub of 1.8' 2026-02-13T02:29:28.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:845: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-13T02:29:28.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:846: TEST_abort_periodic_for_operator: ceph tell 1.8 schedule-deep-scrub 2026-02-13T02:29:28.772 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:29:28.772 INFO:tasks.workunit.client.0.trial112.stdout: "deep": true, 2026-02-13T02:29:28.773 INFO:tasks.workunit.client.0.trial112.stdout: "must": false, 2026-02-13T02:29:28.773 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "2026-02-12T20:05:18.767564+0000" 2026-02-13T02:29:28.773 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:28.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:847: TEST_abort_periodic_for_operator: sleep 1 2026-02-13T02:29:29.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:848: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-13T02:29:29.781 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:850: TEST_abort_periodic_for_operator: seq 1 14 2026-02-13T02:29:29.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:850: TEST_abort_periodic_for_operator: for i in $( seq 1 14 ) 2026-02-13T02:29:29.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:852: TEST_abort_periodic_for_operator: sleep 0.5 2026-02-13T02:29:30.286 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:853: TEST_abort_periodic_for_operator: ceph pg 1.8 query 2026-02-13T02:29:30.286 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:853: TEST_abort_periodic_for_operator: jq .scrubber 2026-02-13T02:29:30.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:853: TEST_abort_periodic_for_operator: stt='{ 2026-02-13T02:29:30.361 INFO:tasks.workunit.client.0.trial112.stderr: "active": true, 2026-02-13T02:29:30.361 INFO:tasks.workunit.client.0.trial112.stderr: "epoch_start": "25", 2026-02-13T02:29:30.361 INFO:tasks.workunit.client.0.trial112.stderr: "start": "1:10000000::::0", 2026-02-13T02:29:30.361 INFO:tasks.workunit.client.0.trial112.stderr: "end": "MIN", 2026-02-13T02:29:30.361 INFO:tasks.workunit.client.0.trial112.stderr: "max_end": "MIN", 2026-02-13T02:29:30.361 INFO:tasks.workunit.client.0.trial112.stderr: "subset_last_update": "0'\''0", 2026-02-13T02:29:30.361 INFO:tasks.workunit.client.0.trial112.stderr: "deep": true, 2026-02-13T02:29:30.361 INFO:tasks.workunit.client.0.trial112.stderr: "req_scrub": false, 2026-02-13T02:29:30.361 INFO:tasks.workunit.client.0.trial112.stderr: "auto_repair": false, 2026-02-13T02:29:30.361 INFO:tasks.workunit.client.0.trial112.stderr: "check_repair": false, 2026-02-13T02:29:30.361 INFO:tasks.workunit.client.0.trial112.stderr: "deep_scrub_on_error": false, 2026-02-13T02:29:30.362 INFO:tasks.workunit.client.0.trial112.stderr: "priority": 5, 2026-02-13T02:29:30.362 INFO:tasks.workunit.client.0.trial112.stderr: "shallow_errors": 0, 2026-02-13T02:29:30.362 INFO:tasks.workunit.client.0.trial112.stderr: "deep_errors": 0, 2026-02-13T02:29:30.362 INFO:tasks.workunit.client.0.trial112.stderr: "fixed": 0, 2026-02-13T02:29:30.362 INFO:tasks.workunit.client.0.trial112.stderr: "waiting_on_whom": [], 2026-02-13T02:29:30.362 INFO:tasks.workunit.client.0.trial112.stderr: "schedule": "scrubbing", 2026-02-13T02:29:30.362 INFO:tasks.workunit.client.0.trial112.stderr: "is_reserving_replicas": false, 2026-02-13T02:29:30.362 INFO:tasks.workunit.client.0.trial112.stderr: "test_sequence": 7553028 2026-02-13T02:29:30.362 INFO:tasks.workunit.client.0.trial112.stderr:}' 2026-02-13T02:29:30.363 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:854: TEST_abort_periodic_for_operator: echo '{' '"active":' true, '"epoch_start":' '"25",' '"start":' '"1:10000000::::0",' '"end":' '"MIN",' '"max_end":' '"MIN",' '"subset_last_update":' '"0'\''0",' '"deep":' true, '"req_scrub":' false, '"auto_repair":' false, '"check_repair":' false, '"deep_scrub_on_error":' false, '"priority":' 5, '"shallow_errors":' 0, '"deep_errors":' 0, '"fixed":' 0, '"waiting_on_whom":' '[],' '"schedule":' '"scrubbing",' '"is_reserving_replicas":' false, '"test_sequence":' 7553028 '}' 2026-02-13T02:29:30.363 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:854: TEST_abort_periodic_for_operator: jq .active 2026-02-13T02:29:30.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:854: TEST_abort_periodic_for_operator: is_active=true 2026-02-13T02:29:30.372 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:855: TEST_abort_periodic_for_operator: echo '{' '"active":' true, '"epoch_start":' '"25",' '"start":' '"1:10000000::::0",' '"end":' '"MIN",' '"max_end":' '"MIN",' '"subset_last_update":' '"0'\''0",' '"deep":' true, '"req_scrub":' false, '"auto_repair":' false, '"check_repair":' false, '"deep_scrub_on_error":' false, '"priority":' 5, '"shallow_errors":' 0, '"deep_errors":' 0, '"fixed":' 0, '"waiting_on_whom":' '[],' '"schedule":' '"scrubbing",' '"is_reserving_replicas":' false, '"test_sequence":' 7553028 '}' 2026-02-13T02:29:30.372 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:855: TEST_abort_periodic_for_operator: jq .is_reserving_replicas 2026-02-13T02:29:30.383 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:855: TEST_abort_periodic_for_operator: is_reserving_replicas=false 2026-02-13T02:29:30.383 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:856: TEST_abort_periodic_for_operator: [[ true = \t\r\u\e ]] 2026-02-13T02:29:30.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:856: TEST_abort_periodic_for_operator: [[ false = \f\a\l\s\e ]] 2026-02-13T02:29:30.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:857: TEST_abort_periodic_for_operator: break 2026-02-13T02:29:30.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:861: TEST_abort_periodic_for_operator: [[ true != \t\r\u\e ]] 2026-02-13T02:29:30.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:861: TEST_abort_periodic_for_operator: [[ false != \f\a\l\s\e ]] 2026-02-13T02:29:30.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:866: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-13T02:29:30.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:873: TEST_abort_periodic_for_operator: (( extr_dbg >= 1 )) 2026-02-13T02:29:30.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:873: TEST_abort_periodic_for_operator: ceph tell 'osd.*' dump_scrub_reservations --format=json-pretty 2026-02-13T02:29:30.446 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T02:29:30.446 INFO:tasks.workunit.client.0.trial112.stdout:osd.0: { 2026-02-13T02:29:30.446 INFO:tasks.workunit.client.0.trial112.stdout: "scrubs_local": 0, 2026-02-13T02:29:30.446 INFO:tasks.workunit.client.0.trial112.stdout: "osd_max_scrubs": 1, 2026-02-13T02:29:30.446 INFO:tasks.workunit.client.0.trial112.stdout: "remote_scrub_reservations": { 2026-02-13T02:29:30.446 INFO:tasks.workunit.client.0.trial112.stdout: "max_allowed": 1, 2026-02-13T02:29:30.446 INFO:tasks.workunit.client.0.trial112.stdout: "min_priority": 0, 2026-02-13T02:29:30.446 INFO:tasks.workunit.client.0.trial112.stdout: "queues": [], 2026-02-13T02:29:30.446 INFO:tasks.workunit.client.0.trial112.stdout: "in_progress": [] 2026-02-13T02:29:30.446 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:29:30.447 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:30.454 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T02:29:30.454 INFO:tasks.workunit.client.0.trial112.stdout:osd.1: { 2026-02-13T02:29:30.454 INFO:tasks.workunit.client.0.trial112.stdout: "scrubs_local": 1, 2026-02-13T02:29:30.454 INFO:tasks.workunit.client.0.trial112.stdout: "osd_max_scrubs": 2, 2026-02-13T02:29:30.454 INFO:tasks.workunit.client.0.trial112.stdout: "remote_scrub_reservations": { 2026-02-13T02:29:30.454 INFO:tasks.workunit.client.0.trial112.stdout: "max_allowed": 2, 2026-02-13T02:29:30.454 INFO:tasks.workunit.client.0.trial112.stdout: "min_priority": 0, 2026-02-13T02:29:30.454 INFO:tasks.workunit.client.0.trial112.stdout: "queues": [], 2026-02-13T02:29:30.454 INFO:tasks.workunit.client.0.trial112.stdout: "in_progress": [] 2026-02-13T02:29:30.455 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:29:30.455 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:30.462 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T02:29:30.462 INFO:tasks.workunit.client.0.trial112.stdout:osd.2: { 2026-02-13T02:29:30.462 INFO:tasks.workunit.client.0.trial112.stdout: "scrubs_local": 0, 2026-02-13T02:29:30.462 INFO:tasks.workunit.client.0.trial112.stdout: "osd_max_scrubs": 1, 2026-02-13T02:29:30.462 INFO:tasks.workunit.client.0.trial112.stdout: "remote_scrub_reservations": { 2026-02-13T02:29:30.462 INFO:tasks.workunit.client.0.trial112.stdout: "max_allowed": 1, 2026-02-13T02:29:30.462 INFO:tasks.workunit.client.0.trial112.stdout: "min_priority": 0, 2026-02-13T02:29:30.462 INFO:tasks.workunit.client.0.trial112.stdout: "queues": [], 2026-02-13T02:29:30.462 INFO:tasks.workunit.client.0.trial112.stdout: "in_progress": [] 2026-02-13T02:29:30.462 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:29:30.462 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:30.470 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T02:29:30.470 INFO:tasks.workunit.client.0.trial112.stdout:osd.3: { 2026-02-13T02:29:30.470 INFO:tasks.workunit.client.0.trial112.stdout: "scrubs_local": 0, 2026-02-13T02:29:30.470 INFO:tasks.workunit.client.0.trial112.stdout: "osd_max_scrubs": 1, 2026-02-13T02:29:30.470 INFO:tasks.workunit.client.0.trial112.stdout: "remote_scrub_reservations": { 2026-02-13T02:29:30.470 INFO:tasks.workunit.client.0.trial112.stdout: "max_allowed": 1, 2026-02-13T02:29:30.470 INFO:tasks.workunit.client.0.trial112.stdout: "min_priority": 0, 2026-02-13T02:29:30.470 INFO:tasks.workunit.client.0.trial112.stdout: "queues": [], 2026-02-13T02:29:30.470 INFO:tasks.workunit.client.0.trial112.stdout: "in_progress": [ 2026-02-13T02:29:30.470 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:30.471 INFO:tasks.workunit.client.0.trial112.stdout: "item": "1.8", 2026-02-13T02:29:30.471 INFO:tasks.workunit.client.0.trial112.stdout: "prio": 0, 2026-02-13T02:29:30.471 INFO:tasks.workunit.client.0.trial112.stdout: "can_preempt": false 2026-02-13T02:29:30.471 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:29:30.471 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:29:30.471 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:29:30.471 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:30.477 INFO:tasks.workunit.client.0.trial112.stdout: 2026-02-13T02:29:30.477 INFO:tasks.workunit.client.0.trial112.stdout:osd.4: { 2026-02-13T02:29:30.477 INFO:tasks.workunit.client.0.trial112.stdout: "scrubs_local": 0, 2026-02-13T02:29:30.478 INFO:tasks.workunit.client.0.trial112.stdout: "osd_max_scrubs": 1, 2026-02-13T02:29:30.478 INFO:tasks.workunit.client.0.trial112.stdout: "remote_scrub_reservations": { 2026-02-13T02:29:30.478 INFO:tasks.workunit.client.0.trial112.stdout: "max_allowed": 1, 2026-02-13T02:29:30.478 INFO:tasks.workunit.client.0.trial112.stdout: "min_priority": 0, 2026-02-13T02:29:30.478 INFO:tasks.workunit.client.0.trial112.stdout: "queues": [], 2026-02-13T02:29:30.478 INFO:tasks.workunit.client.0.trial112.stdout: "in_progress": [ 2026-02-13T02:29:30.478 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:29:30.478 INFO:tasks.workunit.client.0.trial112.stdout: "item": "1.8", 2026-02-13T02:29:30.478 INFO:tasks.workunit.client.0.trial112.stdout: "prio": 0, 2026-02-13T02:29:30.478 INFO:tasks.workunit.client.0.trial112.stdout: "can_preempt": false 2026-02-13T02:29:30.478 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:29:30.478 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:29:30.479 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:29:30.479 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:30.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:876: TEST_abort_periodic_for_operator: set_query_debug 1.9 2026-02-13T02:29:30.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:417: set_query_debug: local pgid=1.9 2026-02-13T02:29:30.486 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:419: set_query_debug: ceph pg dump pgs_brief 2026-02-13T02:29:30.486 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:419: set_query_debug: awk -v 'pg=^1.9' -n -e '$0 ~ pg { print(gensub(/[^0-9]*([0-9]+).*/,"\\1","g",$5)); }' 2026-02-13T02:29:30.632 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs_brief 2026-02-13T02:29:30.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:419: set_query_debug: local prim_osd=1 2026-02-13T02:29:30.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:421: set_query_debug: echo 'Setting scrub debug data. Primary for 1.9 is 1' 2026-02-13T02:29:30.642 INFO:tasks.workunit.client.0.trial112.stdout:Setting scrub debug data. Primary for 1.9 is 1 2026-02-13T02:29:30.643 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:422: set_query_debug: get_asok_path osd.1 2026-02-13T02:29:30.643 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-13T02:29:30.643 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-13T02:29:30.643 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:29:30.643 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:29:30.643 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:29:30.644 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.508946/ceph-osd.1.asok 2026-02-13T02:29:30.644 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:422: set_query_debug: CEPH_ARGS= 2026-02-13T02:29:30.644 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:422: set_query_debug: ceph --format=json daemon /tmp/ceph-asok.508946/ceph-osd.1.asok scrubdebug 1.9 set sessions 2026-02-13T02:29:30.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:877: TEST_abort_periodic_for_operator: ceph tell 1.9 schedule-deep-scrub 2026-02-13T02:29:30.779 INFO:tasks.workunit.client.0.trial112.stdout:{"success":true}{ 2026-02-13T02:29:30.779 INFO:tasks.workunit.client.0.trial112.stdout: "deep": true, 2026-02-13T02:29:30.779 INFO:tasks.workunit.client.0.trial112.stdout: "must": false, 2026-02-13T02:29:30.779 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "2026-02-12T20:05:20.774302+0000" 2026-02-13T02:29:30.779 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:30.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:878: TEST_abort_periodic_for_operator: sleep 0.5 2026-02-13T02:29:31.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:879: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-13T02:29:31.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:880: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-13T02:29:31.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:881: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-13T02:29:31.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:882: TEST_abort_periodic_for_operator: sleep 1 2026-02-13T02:29:32.289 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:883: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-13T02:29:32.289 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:884: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-13T02:29:32.289 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:885: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-13T02:29:32.290 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:888: TEST_abort_periodic_for_operator: ceph pg 1.9 query 2026-02-13T02:29:32.290 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:888: TEST_abort_periodic_for_operator: jq .scrubber 2026-02-13T02:29:32.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:888: TEST_abort_periodic_for_operator: local 'stt2={ 2026-02-13T02:29:32.367 INFO:tasks.workunit.client.0.trial112.stderr: "active": true, 2026-02-13T02:29:32.367 INFO:tasks.workunit.client.0.trial112.stderr: "epoch_start": "25", 2026-02-13T02:29:32.367 INFO:tasks.workunit.client.0.trial112.stderr: "start": "MIN", 2026-02-13T02:29:32.367 INFO:tasks.workunit.client.0.trial112.stderr: "end": "MIN", 2026-02-13T02:29:32.367 INFO:tasks.workunit.client.0.trial112.stderr: "max_end": "MIN", 2026-02-13T02:29:32.367 INFO:tasks.workunit.client.0.trial112.stderr: "subset_last_update": "0'\''0", 2026-02-13T02:29:32.367 INFO:tasks.workunit.client.0.trial112.stderr: "deep": true, 2026-02-13T02:29:32.367 INFO:tasks.workunit.client.0.trial112.stderr: "req_scrub": false, 2026-02-13T02:29:32.368 INFO:tasks.workunit.client.0.trial112.stderr: "auto_repair": false, 2026-02-13T02:29:32.368 INFO:tasks.workunit.client.0.trial112.stderr: "check_repair": false, 2026-02-13T02:29:32.368 INFO:tasks.workunit.client.0.trial112.stderr: "deep_scrub_on_error": false, 2026-02-13T02:29:32.368 INFO:tasks.workunit.client.0.trial112.stderr: "priority": 5, 2026-02-13T02:29:32.368 INFO:tasks.workunit.client.0.trial112.stderr: "shallow_errors": 0, 2026-02-13T02:29:32.368 INFO:tasks.workunit.client.0.trial112.stderr: "deep_errors": 0, 2026-02-13T02:29:32.368 INFO:tasks.workunit.client.0.trial112.stderr: "fixed": 0, 2026-02-13T02:29:32.368 INFO:tasks.workunit.client.0.trial112.stderr: "waiting_on_whom": [], 2026-02-13T02:29:32.368 INFO:tasks.workunit.client.0.trial112.stderr: "schedule": "scrubbing", 2026-02-13T02:29:32.368 INFO:tasks.workunit.client.0.trial112.stderr: "is_reserving_replicas": true, 2026-02-13T02:29:32.368 INFO:tasks.workunit.client.0.trial112.stderr: "osd_to_respond": 4, 2026-02-13T02:29:32.368 INFO:tasks.workunit.client.0.trial112.stderr: "duration_seconds": 2, 2026-02-13T02:29:32.369 INFO:tasks.workunit.client.0.trial112.stderr: "requested_in_order": 2, 2026-02-13T02:29:32.369 INFO:tasks.workunit.client.0.trial112.stderr: "num_to_reserve": 2, 2026-02-13T02:29:32.369 INFO:tasks.workunit.client.0.trial112.stderr: "test_sequence": 7667715 2026-02-13T02:29:32.369 INFO:tasks.workunit.client.0.trial112.stderr:}' 2026-02-13T02:29:32.369 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:889: TEST_abort_periodic_for_operator: local pg2_is_reserving 2026-02-13T02:29:32.369 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:890: TEST_abort_periodic_for_operator: jq .is_reserving_replicas 2026-02-13T02:29:32.369 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:890: TEST_abort_periodic_for_operator: echo '{' '"active":' true, '"epoch_start":' '"25",' '"start":' '"MIN",' '"end":' '"MIN",' '"max_end":' '"MIN",' '"subset_last_update":' '"0'\''0",' '"deep":' true, '"req_scrub":' false, '"auto_repair":' false, '"check_repair":' false, '"deep_scrub_on_error":' false, '"priority":' 5, '"shallow_errors":' 0, '"deep_errors":' 0, '"fixed":' 0, '"waiting_on_whom":' '[],' '"schedule":' '"scrubbing",' '"is_reserving_replicas":' true, '"osd_to_respond":' 4, '"duration_seconds":' 2, '"requested_in_order":' 2, '"num_to_reserve":' 2, '"test_sequence":' 7667715 '}' 2026-02-13T02:29:32.376 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:890: TEST_abort_periodic_for_operator: pg2_is_reserving=true 2026-02-13T02:29:32.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:891: TEST_abort_periodic_for_operator: [[ true != \t\r\u\e ]] 2026-02-13T02:29:32.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:899: TEST_abort_periodic_for_operator: echo 'Instructing 1.9 to perform a high-priority scrub' 2026-02-13T02:29:32.377 INFO:tasks.workunit.client.0.trial112.stdout:Instructing 1.9 to perform a high-priority scrub 2026-02-13T02:29:32.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:900: TEST_abort_periodic_for_operator: ceph tell 1.9 scrub 2026-02-13T02:29:32.441 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:29:32.441 INFO:tasks.workunit.client.0.trial112.stdout: "deep": false, 2026-02-13T02:29:32.441 INFO:tasks.workunit.client.0.trial112.stdout: "must": true, 2026-02-13T02:29:32.441 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "0.000000" 2026-02-13T02:29:32.441 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:29:32.448 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:901: TEST_abort_periodic_for_operator: seq 1 10 2026-02-13T02:29:32.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:901: TEST_abort_periodic_for_operator: for i in $( seq 1 10 ) 2026-02-13T02:29:32.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:903: TEST_abort_periodic_for_operator: sleep 0.5 2026-02-13T02:29:32.952 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:904: TEST_abort_periodic_for_operator: ceph pg 1.9 query 2026-02-13T02:29:32.952 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:904: TEST_abort_periodic_for_operator: jq .scrubber 2026-02-13T02:29:33.023 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:904: TEST_abort_periodic_for_operator: stt2='{ 2026-02-13T02:29:33.023 INFO:tasks.workunit.client.0.trial112.stderr: "active": true, 2026-02-13T02:29:33.023 INFO:tasks.workunit.client.0.trial112.stderr: "epoch_start": "25", 2026-02-13T02:29:33.023 INFO:tasks.workunit.client.0.trial112.stderr: "start": "MIN", 2026-02-13T02:29:33.024 INFO:tasks.workunit.client.0.trial112.stderr: "end": "MIN", 2026-02-13T02:29:33.024 INFO:tasks.workunit.client.0.trial112.stderr: "max_end": "MIN", 2026-02-13T02:29:33.024 INFO:tasks.workunit.client.0.trial112.stderr: "subset_last_update": "0'\''0", 2026-02-13T02:29:33.024 INFO:tasks.workunit.client.0.trial112.stderr: "deep": true, 2026-02-13T02:29:33.024 INFO:tasks.workunit.client.0.trial112.stderr: "req_scrub": false, 2026-02-13T02:29:33.024 INFO:tasks.workunit.client.0.trial112.stderr: "auto_repair": false, 2026-02-13T02:29:33.024 INFO:tasks.workunit.client.0.trial112.stderr: "check_repair": false, 2026-02-13T02:29:33.024 INFO:tasks.workunit.client.0.trial112.stderr: "deep_scrub_on_error": false, 2026-02-13T02:29:33.024 INFO:tasks.workunit.client.0.trial112.stderr: "priority": 0, 2026-02-13T02:29:33.024 INFO:tasks.workunit.client.0.trial112.stderr: "shallow_errors": 0, 2026-02-13T02:29:33.024 INFO:tasks.workunit.client.0.trial112.stderr: "deep_errors": 0, 2026-02-13T02:29:33.024 INFO:tasks.workunit.client.0.trial112.stderr: "fixed": 0, 2026-02-13T02:29:33.024 INFO:tasks.workunit.client.0.trial112.stderr: "waiting_on_whom": [], 2026-02-13T02:29:33.025 INFO:tasks.workunit.client.0.trial112.stderr: "schedule": "scrubbing", 2026-02-13T02:29:33.025 INFO:tasks.workunit.client.0.trial112.stderr: "is_reserving_replicas": false, 2026-02-13T02:29:33.025 INFO:tasks.workunit.client.0.trial112.stderr: "test_sequence": 7667716 2026-02-13T02:29:33.025 INFO:tasks.workunit.client.0.trial112.stderr:}' 2026-02-13T02:29:33.025 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:905: TEST_abort_periodic_for_operator: jq .active 2026-02-13T02:29:33.025 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:905: TEST_abort_periodic_for_operator: echo '{' '"active":' true, '"epoch_start":' '"25",' '"start":' '"MIN",' '"end":' '"MIN",' '"max_end":' '"MIN",' '"subset_last_update":' '"0'\''0",' '"deep":' true, '"req_scrub":' false, '"auto_repair":' false, '"check_repair":' false, '"deep_scrub_on_error":' false, '"priority":' 0, '"shallow_errors":' 0, '"deep_errors":' 0, '"fixed":' 0, '"waiting_on_whom":' '[],' '"schedule":' '"scrubbing",' '"is_reserving_replicas":' false, '"test_sequence":' 7667716 '}' 2026-02-13T02:29:33.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:905: TEST_abort_periodic_for_operator: pg2_is_active=true 2026-02-13T02:29:33.036 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:906: TEST_abort_periodic_for_operator: jq .is_reserving_replicas 2026-02-13T02:29:33.036 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:906: TEST_abort_periodic_for_operator: echo '{' '"active":' true, '"epoch_start":' '"25",' '"start":' '"MIN",' '"end":' '"MIN",' '"max_end":' '"MIN",' '"subset_last_update":' '"0'\''0",' '"deep":' true, '"req_scrub":' false, '"auto_repair":' false, '"check_repair":' false, '"deep_scrub_on_error":' false, '"priority":' 0, '"shallow_errors":' 0, '"deep_errors":' 0, '"fixed":' 0, '"waiting_on_whom":' '[],' '"schedule":' '"scrubbing",' '"is_reserving_replicas":' false, '"test_sequence":' 7667716 '}' 2026-02-13T02:29:33.047 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:906: TEST_abort_periodic_for_operator: pg2_is_reserving=false 2026-02-13T02:29:33.047 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:907: TEST_abort_periodic_for_operator: [[ true = \t\r\u\e ]] 2026-02-13T02:29:33.047 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:907: TEST_abort_periodic_for_operator: [[ false != \t\r\u\e ]] 2026-02-13T02:29:33.047 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:908: TEST_abort_periodic_for_operator: break 2026-02-13T02:29:33.047 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:913: TEST_abort_periodic_for_operator: [[ true != \t\r\u\e ]] 2026-02-13T02:29:33.047 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:913: TEST_abort_periodic_for_operator: [[ false = \t\r\u\e ]] 2026-02-13T02:29:33.048 INFO:tasks.workunit.client.0.trial112.stdout:Done 2026-02-13T02:29:33.048 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:917: TEST_abort_periodic_for_operator: echo Done 2026-02-13T02:29:33.048 INFO:tasks.workunit.client.0.trial112.stdout:-------------- Teardown Test TEST_abort_periodic_for_operator ------------------ 2026-02-13T02:29:33.049 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_abort_periodic_for_operator ------------------' 2026-02-13T02:29:33.050 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2026-02-13T02:29:33.050 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-13T02:29:33.050 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:29:33.050 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-13T02:29:33.050 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:29:33.050 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:29:33.050 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:29:33.050 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:29:33.050 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:29:33.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:29:33.159 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:29:33.160 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:29:33.160 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:29:33.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:29:33.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:29:33.162 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:29:33.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:29:33.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:29:33.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:29:33.164 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:29:33.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:29:33.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:29:33.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-13T02:29:33.190 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:29:33.190 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:29:33.190 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:29:33.190 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.508946 2026-02-13T02:29:33.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:29:33.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:29:33.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_abort_periodic_for_operator ------------------' 2026-02-13T02:29:33.192 INFO:tasks.workunit.client.0.trial112.stdout:-------------- Complete Test TEST_abort_periodic_for_operator ------------------ 2026-02-13T02:29:33.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2026-02-13T02:29:33.193 INFO:tasks.workunit.client.0.trial112.stdout:-------------- Prepare Test TEST_deep_scrub_abort ------------------- 2026-02-13T02:29:33.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_deep_scrub_abort -------------------' 2026-02-13T02:29:33.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2026-02-13T02:29:33.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2026-02-13T02:29:33.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2026-02-13T02:29:33.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-13T02:29:33.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:29:33.195 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-13T02:29:33.195 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:29:33.195 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:29:33.195 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:29:33.195 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:29:33.195 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:29:33.195 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:29:33.196 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:29:33.196 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:29:33.197 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:29:33.198 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:29:33.198 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:29:33.199 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:29:33.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:29:33.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:29:33.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:29:33.200 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:29:33.202 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:29:33.203 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:29:33.203 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-13T02:29:33.204 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:29:33.204 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:29:33.204 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:29:33.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.508946 2026-02-13T02:29:33.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:29:33.206 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:29:33.206 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2026-02-13T02:29:33.207 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T02:29:33.207 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:29:33.207 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:29:33.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.508946 2026-02-13T02:29:33.209 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T02:29:33.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-13T02:29:33.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T02:29:33.210 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2026-02-13T02:29:33.210 INFO:tasks.workunit.client.0.trial112.stdout:-------------- Run Test TEST_deep_scrub_abort ----------------------- 2026-02-13T02:29:33.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_deep_scrub_abort -----------------------' 2026-02-13T02:29:33.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_deep_scrub_abort td/osd-scrub-test 2026-02-13T02:29:33.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:409: TEST_deep_scrub_abort: local dir=td/osd-scrub-test 2026-02-13T02:29:33.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:410: TEST_deep_scrub_abort: _scrub_abort td/osd-scrub-test deep-scrub 2026-02-13T02:29:33.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:293: _scrub_abort: local dir=td/osd-scrub-test 2026-02-13T02:29:33.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:294: _scrub_abort: local poolname=test 2026-02-13T02:29:33.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:295: _scrub_abort: local OSDS=3 2026-02-13T02:29:33.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:296: _scrub_abort: local objects=1000 2026-02-13T02:29:33.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:297: _scrub_abort: local type=deep-scrub 2026-02-13T02:29:33.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:299: _scrub_abort: TESTDATA=testdata.508946 2026-02-13T02:29:33.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:300: _scrub_abort: test deep-scrub = scrub 2026-02-13T02:29:33.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:305: _scrub_abort: stopscrub=nodeep-scrub 2026-02-13T02:29:33.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:306: _scrub_abort: check=nodeep_scrub 2026-02-13T02:29:33.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:309: _scrub_abort: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2026-02-13T02:29:33.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-test 2026-02-13T02:29:33.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-13T02:29:33.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-13T02:29:33.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-13T02:29:33.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-test/a 2026-02-13T02:29:33.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2026-02-13T02:29:33.231 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-13T02:29:33.231 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:29:33.231 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:29:33.232 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:29:33.232 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:29:33.232 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:29:33.232 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:29:33.232 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2026-02-13T02:29:33.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-13T02:29:33.260 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-13T02:29:33.260 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:29:33.260 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:29:33.260 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-13T02:29:33.261 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-13T02:29:33.261 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:29:33.261 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:29:33.261 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:29:33.262 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:29:33.262 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:29:33.262 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:29:33.262 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.508946/ceph-mon.a.asok 2026-02-13T02:29:33.262 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:29:33.262 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.508946/ceph-mon.a.asok config get fsid 2026-02-13T02:29:33.316 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-13T02:29:33.316 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:29:33.316 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:29:33.316 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-13T02:29:33.317 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-13T02:29:33.317 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:29:33.318 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:29:33.318 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:29:33.318 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:29:33.318 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:29:33.318 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:29:33.318 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.508946/ceph-mon.a.asok 2026-02-13T02:29:33.318 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:29:33.318 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.508946/ceph-mon.a.asok config get mon_host 2026-02-13T02:29:33.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:310: _scrub_abort: run_mgr td/osd-scrub-test x --mgr_stats_period=1 2026-02-13T02:29:33.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-test 2026-02-13T02:29:33.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-13T02:29:33.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-13T02:29:33.372 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-13T02:29:33.372 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-test/x 2026-02-13T02:29:33.372 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-13T02:29:33.462 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-13T02:29:33.462 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:29:33.462 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:29:33.463 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:29:33.463 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:29:33.463 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:29:33.463 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:29:33.463 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:29:33.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mgr_stats_period=1 2026-02-13T02:29:33.482 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:311: _scrub_abort: expr 3 - 1 2026-02-13T02:29:33.483 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:311: _scrub_abort: seq 0 2 2026-02-13T02:29:33.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:311: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:29:33.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:317: _scrub_abort: run_osd td/osd-scrub-test 0 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2026-02-13T02:29:33.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-13T02:29:33.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:29:33.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-13T02:29:33.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:29:33.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/0 2026-02-13T02:29:33.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-13T02:29:33.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:29:33.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:29:33.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:29:33.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2026-02-13T02:29:33.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2026-02-13T02:29:33.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:29:33.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:29:33.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-13T02:29:33.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:29:33.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:29:33.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:29:33.488 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:29:33.489 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:29:33.489 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:29:33.489 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:29:33.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:29:33.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:29:33.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:29:33.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:29:33.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-13T02:29:33.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-13T02:29:33.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:29:33.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:29:33.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:29:33.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:29:33.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:29:33.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2026-02-13T02:29:33.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/0 2026-02-13T02:29:33.490 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:29:33.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b42e4b9e-c3b5-42b2-a37e-b1edb8fc4da4 2026-02-13T02:29:33.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 b42e4b9e-c3b5-42b2-a37e-b1edb8fc4da4' 2026-02-13T02:29:33.491 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 b42e4b9e-c3b5-42b2-a37e-b1edb8fc4da4 2026-02-13T02:29:33.491 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:29:33.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCNjI5pLilvHRAAZLHjpLVFpbMxhkib1t64dQ== 2026-02-13T02:29:33.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCNjI5pLilvHRAAZLHjpLVFpbMxhkib1t64dQ=="}' 2026-02-13T02:29:33.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b42e4b9e-c3b5-42b2-a37e-b1edb8fc4da4 -i td/osd-scrub-test/0/new.json 2026-02-13T02:29:33.591 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:29:33.599 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/0/new.json 2026-02-13T02:29:33.600 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQCNjI5pLilvHRAAZLHjpLVFpbMxhkib1t64dQ== --osd-uuid b42e4b9e-c3b5-42b2-a37e-b1edb8fc4da4 2026-02-13T02:29:33.617 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:29:33.610+0000 7f99ff754a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:29:33.619 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:29:33.610+0000 7f99ff754a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:29:33.620 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:29:33.614+0000 7f99ff754a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:29:33.620 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:29:33.614+0000 7f99ff754a40 -1 bdev(0x5616c549ec00 td/osd-scrub-test/0/block) open stat got: (1) Operation not permitted 2026-02-13T02:29:33.620 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:29:33.614+0000 7f99ff754a40 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2026-02-13T02:29:34.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2026-02-13T02:29:34.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:29:34.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-13T02:29:34.365 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T02:29:34.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:29:34.458 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T02:29:34.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-13T02:29:34.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2026-02-13T02:29:34.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:29:34.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:29:34.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:29:34.485 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:29:34.478+0000 7f7d919f0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:29:34.487 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:29:34.478+0000 7f7d919f0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:29:34.489 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:29:34.482+0000 7f7d919f0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:29:34.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-13T02:29:34.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:29:34.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:29:34.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:29:34.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:29:34.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:29:34.550 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:29:34.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:29:34.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:29:34.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:29:34.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:29:34.893 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:29:34.886+0000 7f7d919f0a40 -1 Falling back to public interface 2026-02-13T02:29:35.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:29:35.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:29:35.641 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:29:35.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:29:35.644 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:29:35.644 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:29:35.735 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:29:35.886 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:29:35.878+0000 7f7d919f0a40 -1 osd.0 0 log_to_monitors true 2026-02-13T02:29:36.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:29:36.737 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:29:36.737 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:29:36.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:29:36.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:29:36.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:29:36.891 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2368047041,v1:127.0.0.1:6801/2368047041] [v2:127.0.0.1:6802/2368047041,v1:127.0.0.1:6803/2368047041] exists,up b42e4b9e-c3b5-42b2-a37e-b1edb8fc4da4 2026-02-13T02:29:36.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:29:36.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:29:36.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:29:36.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:311: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:29:36.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:317: _scrub_abort: run_osd td/osd-scrub-test 1 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2026-02-13T02:29:36.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-13T02:29:36.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:29:36.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-13T02:29:36.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:29:36.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/1 2026-02-13T02:29:36.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-13T02:29:36.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:29:36.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:29:36.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:29:36.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2026-02-13T02:29:36.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2026-02-13T02:29:36.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:29:36.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:29:36.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-13T02:29:36.896 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:29:36.896 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:29:36.896 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:29:36.896 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:29:36.896 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:29:36.896 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:29:36.896 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:29:36.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:29:36.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:29:36.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:29:36.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:29:36.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-13T02:29:36.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-13T02:29:36.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:29:36.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:29:36.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:29:36.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:29:36.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:29:36.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2026-02-13T02:29:36.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/1 2026-02-13T02:29:36.898 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:29:36.898 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e1823648-96a2-4df6-af41-ebb224e1109c 2026-02-13T02:29:36.898 INFO:tasks.workunit.client.0.trial112.stdout:add osd1 e1823648-96a2-4df6-af41-ebb224e1109c 2026-02-13T02:29:36.899 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 e1823648-96a2-4df6-af41-ebb224e1109c' 2026-02-13T02:29:36.899 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:29:36.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCQjI5py9+6NRAAwdUdGL30uuCRlpVA8pYV2w== 2026-02-13T02:29:36.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCQjI5py9+6NRAAwdUdGL30uuCRlpVA8pYV2w=="}' 2026-02-13T02:29:36.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e1823648-96a2-4df6-af41-ebb224e1109c -i td/osd-scrub-test/1/new.json 2026-02-13T02:29:37.062 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:29:37.072 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/1/new.json 2026-02-13T02:29:37.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQCQjI5py9+6NRAAwdUdGL30uuCRlpVA8pYV2w== --osd-uuid e1823648-96a2-4df6-af41-ebb224e1109c 2026-02-13T02:29:37.090 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:29:37.082+0000 7f48141f7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:29:37.091 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:29:37.086+0000 7f48141f7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:29:37.093 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:29:37.086+0000 7f48141f7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:29:37.093 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:29:37.086+0000 7f48141f7a40 -1 bdev(0x55781615ac00 td/osd-scrub-test/1/block) open stat got: (1) Operation not permitted 2026-02-13T02:29:37.093 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:29:37.086+0000 7f48141f7a40 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2026-02-13T02:29:37.779 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2026-02-13T02:29:37.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:29:37.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-13T02:29:37.781 INFO:tasks.workunit.client.0.trial112.stdout:adding osd1 key to auth repository 2026-02-13T02:29:37.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:29:37.973 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T02:29:37.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-13T02:29:37.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2026-02-13T02:29:37.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:29:37.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:29:37.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:29:37.988 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:29:37.982+0000 7f1092910a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:29:37.990 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:29:37.982+0000 7f1092910a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:29:37.990 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:29:37.982+0000 7f1092910a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:29:38.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-13T02:29:38.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:29:38.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:29:38.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:29:38.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:29:38.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:29:38.140 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:29:38.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:29:38.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:29:38.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:29:38.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:29:38.349 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:29:38.342+0000 7f1092910a40 -1 Falling back to public interface 2026-02-13T02:29:38.945 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:29:38.938+0000 7f1092910a40 -1 osd.1 0 log_to_monitors true 2026-02-13T02:29:39.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:29:39.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:29:39.299 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:29:39.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:29:39.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:29:39.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:29:39.457 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3354139744,v1:127.0.0.1:6811/3354139744] [v2:127.0.0.1:6812/3354139744,v1:127.0.0.1:6813/3354139744] exists,up e1823648-96a2-4df6-af41-ebb224e1109c 2026-02-13T02:29:39.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:29:39.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:29:39.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:29:39.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:311: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:29:39.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:317: _scrub_abort: run_osd td/osd-scrub-test 2 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2026-02-13T02:29:39.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-13T02:29:39.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:29:39.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-13T02:29:39.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:29:39.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/2 2026-02-13T02:29:39.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-13T02:29:39.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:29:39.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:29:39.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:29:39.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2026-02-13T02:29:39.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2026-02-13T02:29:39.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:29:39.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:29:39.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-13T02:29:39.462 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:29:39.462 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:29:39.462 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:29:39.462 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:29:39.462 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:29:39.462 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:29:39.462 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:29:39.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:29:39.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:29:39.463 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:29:39.463 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:29:39.463 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-13T02:29:39.463 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-13T02:29:39.463 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:29:39.463 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:29:39.463 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:29:39.463 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:29:39.463 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:29:39.463 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2026-02-13T02:29:39.463 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/2 2026-02-13T02:29:39.463 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:29:39.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f45a7882-9275-42db-8623-6e8bc5eaa649 2026-02-13T02:29:39.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 f45a7882-9275-42db-8623-6e8bc5eaa649' 2026-02-13T02:29:39.464 INFO:tasks.workunit.client.0.trial112.stdout:add osd2 f45a7882-9275-42db-8623-6e8bc5eaa649 2026-02-13T02:29:39.465 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:29:39.474 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCTjI5pCGvrGxAArmaZmLPfTXNrhXQOMl9T5w== 2026-02-13T02:29:39.474 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCTjI5pCGvrGxAArmaZmLPfTXNrhXQOMl9T5w=="}' 2026-02-13T02:29:39.474 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f45a7882-9275-42db-8623-6e8bc5eaa649 -i td/osd-scrub-test/2/new.json 2026-02-13T02:29:39.637 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:29:39.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/2/new.json 2026-02-13T02:29:39.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQCTjI5pCGvrGxAArmaZmLPfTXNrhXQOMl9T5w== --osd-uuid f45a7882-9275-42db-8623-6e8bc5eaa649 2026-02-13T02:29:39.665 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:29:39.658+0000 7f7973de6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:29:39.666 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:29:39.658+0000 7f7973de6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:29:39.668 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:29:39.662+0000 7f7973de6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:29:39.668 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:29:39.662+0000 7f7973de6a40 -1 bdev(0x56017b2e0c00 td/osd-scrub-test/2/block) open stat got: (1) Operation not permitted 2026-02-13T02:29:39.669 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:29:39.662+0000 7f7973de6a40 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2026-02-13T02:29:40.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2026-02-13T02:29:40.319 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:29:40.319 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-13T02:29:40.320 INFO:tasks.workunit.client.0.trial112.stdout:adding osd2 key to auth repository 2026-02-13T02:29:40.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:29:40.511 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-13T02:29:40.512 INFO:tasks.workunit.client.0.trial112.stdout:start osd.2 2026-02-13T02:29:40.512 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2026-02-13T02:29:40.512 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:29:40.512 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:29:40.513 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:29:40.536 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:29:40.530+0000 7f10a0b6aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:29:40.538 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:29:40.530+0000 7f10a0b6aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:29:40.539 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:29:40.534+0000 7f10a0b6aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:29:40.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-13T02:29:40.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:29:40.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T02:29:40.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:29:40.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:29:40.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:29:40.676 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:29:40.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:29:40.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:29:40.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:29:40.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:29:40.905 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:29:40.898+0000 7f10a0b6aa40 -1 Falling back to public interface 2026-02-13T02:29:41.446 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:29:41.438+0000 7f10a0b6aa40 -1 osd.2 0 log_to_monitors true 2026-02-13T02:29:41.839 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:29:41.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:29:41.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:29:41.840 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:29:41.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:29:41.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:29:42.001 INFO:tasks.workunit.client.0.trial112.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1683150994,v1:127.0.0.1:6819/1683150994] [v2:127.0.0.1:6820/1683150994,v1:127.0.0.1:6821/1683150994] exists,up f45a7882-9275-42db-8623-6e8bc5eaa649 2026-02-13T02:29:42.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:29:42.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:29:42.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:29:42.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:325: _scrub_abort: create_pool test 1 1 2026-02-13T02:29:42.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2026-02-13T02:29:42.209 INFO:tasks.workunit.client.0.trial112.stderr:pool 'test' created 2026-02-13T02:29:42.219 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:29:43.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:326: _scrub_abort: wait_for_clean 2026-02-13T02:29:43.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:29:43.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:29:43.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:29:43.222 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:29:43.223 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:29:43.223 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:29:43.223 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:29:43.223 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:29:43.223 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:29:43.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:29:43.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:29:43.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:29:43.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:29:43.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:29:43.329 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:29:43.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:29:43.489 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:29:43.489 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:29:43.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:29:43.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:29:43.490 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:29:43.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836488 2026-02-13T02:29:43.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836488 2026-02-13T02:29:43.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488' 2026-02-13T02:29:43.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:29:43.565 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:29:43.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705669 2026-02-13T02:29:43.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705669 2026-02-13T02:29:43.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-38654705669' 2026-02-13T02:29:43.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:29:43.637 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:29:43.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542147 2026-02-13T02:29:43.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542147 2026-02-13T02:29:43.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-38654705669 2-60129542147' 2026-02-13T02:29:43.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:29:43.714 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836488 2026-02-13T02:29:43.714 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:29:43.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:29:43.715 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836488 2026-02-13T02:29:43.715 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:29:43.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836488 2026-02-13T02:29:43.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2026-02-13T02:29:43.717 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836488 2026-02-13T02:29:43.717 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:29:43.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836488 2026-02-13T02:29:43.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:29:43.874 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705669 2026-02-13T02:29:43.874 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:29:43.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:29:43.876 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705669 2026-02-13T02:29:43.876 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:29:43.877 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705669 2026-02-13T02:29:43.878 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705669 2026-02-13T02:29:43.878 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705669' 2026-02-13T02:29:43.878 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:29:44.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705669 -lt 38654705669 2026-02-13T02:29:44.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:29:44.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542147 2026-02-13T02:29:44.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:29:44.048 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:29:44.049 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542147 2026-02-13T02:29:44.049 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:29:44.050 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542147 2026-02-13T02:29:44.050 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542147' 2026-02-13T02:29:44.050 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542147 2026-02-13T02:29:44.051 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:29:44.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542146 -lt 60129542147 2026-02-13T02:29:44.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:29:45.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:29:45.217 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:29:45.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542146 -lt 60129542147 2026-02-13T02:29:45.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:29:46.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:29:46.381 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:29:46.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542149 -lt 60129542147 2026-02-13T02:29:46.540 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:29:46.540 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:29:46.540 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:29:46.729 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:29:46.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:29:46.730 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:29:46.730 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:29:46.730 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:29:46.730 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:29:46.731 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:29:46.731 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:29:46.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:29:46.887 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:29:46.888 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:29:46.888 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:29:47.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:29:47.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:29:47.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:29:47.081 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:327: _scrub_abort: ceph osd dump 2026-02-13T02:29:47.081 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:327: _scrub_abort: grep '^pool.*['\'']test['\'']' 2026-02-13T02:29:47.081 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:327: _scrub_abort: awk '{ print $2 }' 2026-02-13T02:29:47.242 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:327: _scrub_abort: poolid=1 2026-02-13T02:29:47.242 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:329: _scrub_abort: dd if=/dev/urandom of=testdata.508946 bs=1032 count=1 2026-02-13T02:29:47.243 INFO:tasks.workunit.client.0.trial112.stderr:1+0 records in 2026-02-13T02:29:47.243 INFO:tasks.workunit.client.0.trial112.stderr:1+0 records out 2026-02-13T02:29:47.243 INFO:tasks.workunit.client.0.trial112.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 5.2708e-05 s, 19.6 MB/s 2026-02-13T02:29:47.243 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: seq 1 1000 2026-02-13T02:29:47.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:47.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj1 testdata.508946 2026-02-13T02:29:47.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:47.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj2 testdata.508946 2026-02-13T02:29:47.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:47.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj3 testdata.508946 2026-02-13T02:29:47.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:47.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj4 testdata.508946 2026-02-13T02:29:47.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:47.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj5 testdata.508946 2026-02-13T02:29:47.361 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:47.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj6 testdata.508946 2026-02-13T02:29:47.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:47.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj7 testdata.508946 2026-02-13T02:29:47.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:47.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj8 testdata.508946 2026-02-13T02:29:47.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:47.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj9 testdata.508946 2026-02-13T02:29:47.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:47.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj10 testdata.508946 2026-02-13T02:29:47.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:47.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj11 testdata.508946 2026-02-13T02:29:47.512 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:47.512 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj12 testdata.508946 2026-02-13T02:29:47.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:47.536 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj13 testdata.508946 2026-02-13T02:29:47.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:47.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj14 testdata.508946 2026-02-13T02:29:47.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:47.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj15 testdata.508946 2026-02-13T02:29:47.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:47.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj16 testdata.508946 2026-02-13T02:29:47.627 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:47.627 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj17 testdata.508946 2026-02-13T02:29:47.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:47.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj18 testdata.508946 2026-02-13T02:29:47.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:47.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj19 testdata.508946 2026-02-13T02:29:47.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:47.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj20 testdata.508946 2026-02-13T02:29:47.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:47.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj21 testdata.508946 2026-02-13T02:29:47.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:47.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj22 testdata.508946 2026-02-13T02:29:47.766 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:47.766 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj23 testdata.508946 2026-02-13T02:29:47.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:47.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj24 testdata.508946 2026-02-13T02:29:47.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:47.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj25 testdata.508946 2026-02-13T02:29:47.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:47.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj26 testdata.508946 2026-02-13T02:29:47.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:47.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj27 testdata.508946 2026-02-13T02:29:47.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:47.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj28 testdata.508946 2026-02-13T02:29:47.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:47.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj29 testdata.508946 2026-02-13T02:29:47.926 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:47.926 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj30 testdata.508946 2026-02-13T02:29:47.944 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:47.944 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj31 testdata.508946 2026-02-13T02:29:47.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:47.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj32 testdata.508946 2026-02-13T02:29:47.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:47.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj33 testdata.508946 2026-02-13T02:29:48.015 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.015 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj34 testdata.508946 2026-02-13T02:29:48.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj35 testdata.508946 2026-02-13T02:29:48.056 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.056 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj36 testdata.508946 2026-02-13T02:29:48.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj37 testdata.508946 2026-02-13T02:29:48.105 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.105 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj38 testdata.508946 2026-02-13T02:29:48.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj39 testdata.508946 2026-02-13T02:29:48.155 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.155 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj40 testdata.508946 2026-02-13T02:29:48.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj41 testdata.508946 2026-02-13T02:29:48.195 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.195 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj42 testdata.508946 2026-02-13T02:29:48.219 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.219 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj43 testdata.508946 2026-02-13T02:29:48.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj44 testdata.508946 2026-02-13T02:29:48.269 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.269 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj45 testdata.508946 2026-02-13T02:29:48.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj46 testdata.508946 2026-02-13T02:29:48.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj47 testdata.508946 2026-02-13T02:29:48.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj48 testdata.508946 2026-02-13T02:29:48.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj49 testdata.508946 2026-02-13T02:29:48.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj50 testdata.508946 2026-02-13T02:29:48.416 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.416 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj51 testdata.508946 2026-02-13T02:29:48.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj52 testdata.508946 2026-02-13T02:29:48.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj53 testdata.508946 2026-02-13T02:29:48.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj54 testdata.508946 2026-02-13T02:29:48.507 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.508 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj55 testdata.508946 2026-02-13T02:29:48.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj56 testdata.508946 2026-02-13T02:29:48.557 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.557 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj57 testdata.508946 2026-02-13T02:29:48.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj58 testdata.508946 2026-02-13T02:29:48.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj59 testdata.508946 2026-02-13T02:29:48.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj60 testdata.508946 2026-02-13T02:29:48.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj61 testdata.508946 2026-02-13T02:29:48.681 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj62 testdata.508946 2026-02-13T02:29:48.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj63 testdata.508946 2026-02-13T02:29:48.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj64 testdata.508946 2026-02-13T02:29:48.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj65 testdata.508946 2026-02-13T02:29:48.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.762 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj66 testdata.508946 2026-02-13T02:29:48.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj67 testdata.508946 2026-02-13T02:29:48.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj68 testdata.508946 2026-02-13T02:29:48.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj69 testdata.508946 2026-02-13T02:29:48.859 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.859 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj70 testdata.508946 2026-02-13T02:29:48.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj71 testdata.508946 2026-02-13T02:29:48.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj72 testdata.508946 2026-02-13T02:29:48.926 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.926 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj73 testdata.508946 2026-02-13T02:29:48.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj74 testdata.508946 2026-02-13T02:29:48.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj75 testdata.508946 2026-02-13T02:29:48.998 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:48.998 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj76 testdata.508946 2026-02-13T02:29:49.023 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.023 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj77 testdata.508946 2026-02-13T02:29:49.047 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.047 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj78 testdata.508946 2026-02-13T02:29:49.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj79 testdata.508946 2026-02-13T02:29:49.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj80 testdata.508946 2026-02-13T02:29:49.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj81 testdata.508946 2026-02-13T02:29:49.144 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.144 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj82 testdata.508946 2026-02-13T02:29:49.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj83 testdata.508946 2026-02-13T02:29:49.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj84 testdata.508946 2026-02-13T02:29:49.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj85 testdata.508946 2026-02-13T02:29:49.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj86 testdata.508946 2026-02-13T02:29:49.254 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.254 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj87 testdata.508946 2026-02-13T02:29:49.277 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.277 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj88 testdata.508946 2026-02-13T02:29:49.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj89 testdata.508946 2026-02-13T02:29:49.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj90 testdata.508946 2026-02-13T02:29:49.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj91 testdata.508946 2026-02-13T02:29:49.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj92 testdata.508946 2026-02-13T02:29:49.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj93 testdata.508946 2026-02-13T02:29:49.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj94 testdata.508946 2026-02-13T02:29:49.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj95 testdata.508946 2026-02-13T02:29:49.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj96 testdata.508946 2026-02-13T02:29:49.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj97 testdata.508946 2026-02-13T02:29:49.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj98 testdata.508946 2026-02-13T02:29:49.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj99 testdata.508946 2026-02-13T02:29:49.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj100 testdata.508946 2026-02-13T02:29:49.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj101 testdata.508946 2026-02-13T02:29:49.619 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.619 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj102 testdata.508946 2026-02-13T02:29:49.644 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.644 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj103 testdata.508946 2026-02-13T02:29:49.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj104 testdata.508946 2026-02-13T02:29:49.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj105 testdata.508946 2026-02-13T02:29:49.718 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.718 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj106 testdata.508946 2026-02-13T02:29:49.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj107 testdata.508946 2026-02-13T02:29:49.767 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.767 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj108 testdata.508946 2026-02-13T02:29:49.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj109 testdata.508946 2026-02-13T02:29:49.816 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.816 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj110 testdata.508946 2026-02-13T02:29:49.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj111 testdata.508946 2026-02-13T02:29:49.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj112 testdata.508946 2026-02-13T02:29:49.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj113 testdata.508946 2026-02-13T02:29:49.914 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.914 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj114 testdata.508946 2026-02-13T02:29:49.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj115 testdata.508946 2026-02-13T02:29:49.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj116 testdata.508946 2026-02-13T02:29:49.988 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:49.988 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj117 testdata.508946 2026-02-13T02:29:50.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj118 testdata.508946 2026-02-13T02:29:50.037 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.037 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj119 testdata.508946 2026-02-13T02:29:50.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj120 testdata.508946 2026-02-13T02:29:50.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj121 testdata.508946 2026-02-13T02:29:50.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj122 testdata.508946 2026-02-13T02:29:50.136 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.136 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj123 testdata.508946 2026-02-13T02:29:50.160 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.160 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj124 testdata.508946 2026-02-13T02:29:50.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj125 testdata.508946 2026-02-13T02:29:50.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj126 testdata.508946 2026-02-13T02:29:50.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj127 testdata.508946 2026-02-13T02:29:50.251 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.251 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj128 testdata.508946 2026-02-13T02:29:50.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj129 testdata.508946 2026-02-13T02:29:50.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj130 testdata.508946 2026-02-13T02:29:50.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj131 testdata.508946 2026-02-13T02:29:50.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj132 testdata.508946 2026-02-13T02:29:50.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj133 testdata.508946 2026-02-13T02:29:50.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj134 testdata.508946 2026-02-13T02:29:50.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj135 testdata.508946 2026-02-13T02:29:50.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj136 testdata.508946 2026-02-13T02:29:50.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj137 testdata.508946 2026-02-13T02:29:50.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj138 testdata.508946 2026-02-13T02:29:50.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj139 testdata.508946 2026-02-13T02:29:50.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj140 testdata.508946 2026-02-13T02:29:50.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj141 testdata.508946 2026-02-13T02:29:50.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj142 testdata.508946 2026-02-13T02:29:50.613 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.613 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj143 testdata.508946 2026-02-13T02:29:50.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.639 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj144 testdata.508946 2026-02-13T02:29:50.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj145 testdata.508946 2026-02-13T02:29:50.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj146 testdata.508946 2026-02-13T02:29:50.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj147 testdata.508946 2026-02-13T02:29:50.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj148 testdata.508946 2026-02-13T02:29:50.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj149 testdata.508946 2026-02-13T02:29:50.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj150 testdata.508946 2026-02-13T02:29:50.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj151 testdata.508946 2026-02-13T02:29:50.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj152 testdata.508946 2026-02-13T02:29:50.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj153 testdata.508946 2026-02-13T02:29:50.877 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.878 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj154 testdata.508946 2026-02-13T02:29:50.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj155 testdata.508946 2026-02-13T02:29:50.926 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.926 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj156 testdata.508946 2026-02-13T02:29:50.944 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.944 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj157 testdata.508946 2026-02-13T02:29:50.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj158 testdata.508946 2026-02-13T02:29:50.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:50.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj159 testdata.508946 2026-02-13T02:29:51.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj160 testdata.508946 2026-02-13T02:29:51.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj161 testdata.508946 2026-02-13T02:29:51.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj162 testdata.508946 2026-02-13T02:29:51.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj163 testdata.508946 2026-02-13T02:29:51.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj164 testdata.508946 2026-02-13T02:29:51.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj165 testdata.508946 2026-02-13T02:29:51.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj166 testdata.508946 2026-02-13T02:29:51.183 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.183 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj167 testdata.508946 2026-02-13T02:29:51.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj168 testdata.508946 2026-02-13T02:29:51.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj169 testdata.508946 2026-02-13T02:29:51.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj170 testdata.508946 2026-02-13T02:29:51.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj171 testdata.508946 2026-02-13T02:29:51.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj172 testdata.508946 2026-02-13T02:29:51.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj173 testdata.508946 2026-02-13T02:29:51.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.343 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj174 testdata.508946 2026-02-13T02:29:51.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj175 testdata.508946 2026-02-13T02:29:51.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj176 testdata.508946 2026-02-13T02:29:51.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj177 testdata.508946 2026-02-13T02:29:51.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj178 testdata.508946 2026-02-13T02:29:51.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj179 testdata.508946 2026-02-13T02:29:51.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj180 testdata.508946 2026-02-13T02:29:51.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj181 testdata.508946 2026-02-13T02:29:51.523 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj182 testdata.508946 2026-02-13T02:29:51.548 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.548 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj183 testdata.508946 2026-02-13T02:29:51.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj184 testdata.508946 2026-02-13T02:29:51.597 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.597 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj185 testdata.508946 2026-02-13T02:29:51.621 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.621 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj186 testdata.508946 2026-02-13T02:29:51.646 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.646 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj187 testdata.508946 2026-02-13T02:29:51.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj188 testdata.508946 2026-02-13T02:29:51.694 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.694 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj189 testdata.508946 2026-02-13T02:29:51.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj190 testdata.508946 2026-02-13T02:29:51.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj191 testdata.508946 2026-02-13T02:29:51.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj192 testdata.508946 2026-02-13T02:29:51.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj193 testdata.508946 2026-02-13T02:29:51.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj194 testdata.508946 2026-02-13T02:29:51.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj195 testdata.508946 2026-02-13T02:29:51.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj196 testdata.508946 2026-02-13T02:29:51.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj197 testdata.508946 2026-02-13T02:29:51.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj198 testdata.508946 2026-02-13T02:29:51.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj199 testdata.508946 2026-02-13T02:29:51.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj200 testdata.508946 2026-02-13T02:29:51.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:51.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj201 testdata.508946 2026-02-13T02:29:52.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj202 testdata.508946 2026-02-13T02:29:52.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj203 testdata.508946 2026-02-13T02:29:52.057 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.057 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj204 testdata.508946 2026-02-13T02:29:52.082 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.082 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj205 testdata.508946 2026-02-13T02:29:52.107 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.107 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj206 testdata.508946 2026-02-13T02:29:52.132 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.132 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj207 testdata.508946 2026-02-13T02:29:52.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj208 testdata.508946 2026-02-13T02:29:52.181 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.181 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj209 testdata.508946 2026-02-13T02:29:52.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.206 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj210 testdata.508946 2026-02-13T02:29:52.230 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.230 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj211 testdata.508946 2026-02-13T02:29:52.255 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.255 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj212 testdata.508946 2026-02-13T02:29:52.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj213 testdata.508946 2026-02-13T02:29:52.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj214 testdata.508946 2026-02-13T02:29:52.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj215 testdata.508946 2026-02-13T02:29:52.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj216 testdata.508946 2026-02-13T02:29:52.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj217 testdata.508946 2026-02-13T02:29:52.402 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.402 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj218 testdata.508946 2026-02-13T02:29:52.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj219 testdata.508946 2026-02-13T02:29:52.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj220 testdata.508946 2026-02-13T02:29:52.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj221 testdata.508946 2026-02-13T02:29:52.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj222 testdata.508946 2026-02-13T02:29:52.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj223 testdata.508946 2026-02-13T02:29:52.536 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.536 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj224 testdata.508946 2026-02-13T02:29:52.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj225 testdata.508946 2026-02-13T02:29:52.576 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.576 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj226 testdata.508946 2026-02-13T02:29:52.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj227 testdata.508946 2026-02-13T02:29:52.627 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.627 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj228 testdata.508946 2026-02-13T02:29:52.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj229 testdata.508946 2026-02-13T02:29:52.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj230 testdata.508946 2026-02-13T02:29:52.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj231 testdata.508946 2026-02-13T02:29:52.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj232 testdata.508946 2026-02-13T02:29:52.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj233 testdata.508946 2026-02-13T02:29:52.775 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.775 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj234 testdata.508946 2026-02-13T02:29:52.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj235 testdata.508946 2026-02-13T02:29:52.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj236 testdata.508946 2026-02-13T02:29:52.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj237 testdata.508946 2026-02-13T02:29:52.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj238 testdata.508946 2026-02-13T02:29:52.899 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.899 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj239 testdata.508946 2026-02-13T02:29:52.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj240 testdata.508946 2026-02-13T02:29:52.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj241 testdata.508946 2026-02-13T02:29:52.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj242 testdata.508946 2026-02-13T02:29:52.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:52.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj243 testdata.508946 2026-02-13T02:29:53.016 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.017 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj244 testdata.508946 2026-02-13T02:29:53.042 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.042 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj245 testdata.508946 2026-02-13T02:29:53.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj246 testdata.508946 2026-02-13T02:29:53.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj247 testdata.508946 2026-02-13T02:29:53.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj248 testdata.508946 2026-02-13T02:29:53.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj249 testdata.508946 2026-02-13T02:29:53.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj250 testdata.508946 2026-02-13T02:29:53.190 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.190 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj251 testdata.508946 2026-02-13T02:29:53.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj252 testdata.508946 2026-02-13T02:29:53.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj253 testdata.508946 2026-02-13T02:29:53.258 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.258 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj254 testdata.508946 2026-02-13T02:29:53.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj255 testdata.508946 2026-02-13T02:29:53.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj256 testdata.508946 2026-02-13T02:29:53.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj257 testdata.508946 2026-02-13T02:29:53.347 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.347 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj258 testdata.508946 2026-02-13T02:29:53.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj259 testdata.508946 2026-02-13T02:29:53.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj260 testdata.508946 2026-02-13T02:29:53.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj261 testdata.508946 2026-02-13T02:29:53.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj262 testdata.508946 2026-02-13T02:29:53.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj263 testdata.508946 2026-02-13T02:29:53.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj264 testdata.508946 2026-02-13T02:29:53.513 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.513 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj265 testdata.508946 2026-02-13T02:29:53.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj266 testdata.508946 2026-02-13T02:29:53.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj267 testdata.508946 2026-02-13T02:29:53.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj268 testdata.508946 2026-02-13T02:29:53.613 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.613 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj269 testdata.508946 2026-02-13T02:29:53.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj270 testdata.508946 2026-02-13T02:29:53.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj271 testdata.508946 2026-02-13T02:29:53.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj272 testdata.508946 2026-02-13T02:29:53.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj273 testdata.508946 2026-02-13T02:29:53.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj274 testdata.508946 2026-02-13T02:29:53.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj275 testdata.508946 2026-02-13T02:29:53.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj276 testdata.508946 2026-02-13T02:29:53.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj277 testdata.508946 2026-02-13T02:29:53.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj278 testdata.508946 2026-02-13T02:29:53.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.854 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj279 testdata.508946 2026-02-13T02:29:53.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj280 testdata.508946 2026-02-13T02:29:53.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj281 testdata.508946 2026-02-13T02:29:53.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj282 testdata.508946 2026-02-13T02:29:53.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj283 testdata.508946 2026-02-13T02:29:53.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj284 testdata.508946 2026-02-13T02:29:53.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:53.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj285 testdata.508946 2026-02-13T02:29:54.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj286 testdata.508946 2026-02-13T02:29:54.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj287 testdata.508946 2026-02-13T02:29:54.056 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.057 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj288 testdata.508946 2026-02-13T02:29:54.074 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.074 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj289 testdata.508946 2026-02-13T02:29:54.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj290 testdata.508946 2026-02-13T02:29:54.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj291 testdata.508946 2026-02-13T02:29:54.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj292 testdata.508946 2026-02-13T02:29:54.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj293 testdata.508946 2026-02-13T02:29:54.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj294 testdata.508946 2026-02-13T02:29:54.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj295 testdata.508946 2026-02-13T02:29:54.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj296 testdata.508946 2026-02-13T02:29:54.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj297 testdata.508946 2026-02-13T02:29:54.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj298 testdata.508946 2026-02-13T02:29:54.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj299 testdata.508946 2026-02-13T02:29:54.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj300 testdata.508946 2026-02-13T02:29:54.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj301 testdata.508946 2026-02-13T02:29:54.358 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.358 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj302 testdata.508946 2026-02-13T02:29:54.382 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.382 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj303 testdata.508946 2026-02-13T02:29:54.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj304 testdata.508946 2026-02-13T02:29:54.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj305 testdata.508946 2026-02-13T02:29:54.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj306 testdata.508946 2026-02-13T02:29:54.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj307 testdata.508946 2026-02-13T02:29:54.498 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.498 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj308 testdata.508946 2026-02-13T02:29:54.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj309 testdata.508946 2026-02-13T02:29:54.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj310 testdata.508946 2026-02-13T02:29:54.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj311 testdata.508946 2026-02-13T02:29:54.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj312 testdata.508946 2026-02-13T02:29:54.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj313 testdata.508946 2026-02-13T02:29:54.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj314 testdata.508946 2026-02-13T02:29:54.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj315 testdata.508946 2026-02-13T02:29:54.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj316 testdata.508946 2026-02-13T02:29:54.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj317 testdata.508946 2026-02-13T02:29:54.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj318 testdata.508946 2026-02-13T02:29:54.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj319 testdata.508946 2026-02-13T02:29:54.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj320 testdata.508946 2026-02-13T02:29:54.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj321 testdata.508946 2026-02-13T02:29:54.823 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.823 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj322 testdata.508946 2026-02-13T02:29:54.848 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.848 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj323 testdata.508946 2026-02-13T02:29:54.870 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.870 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj324 testdata.508946 2026-02-13T02:29:54.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj325 testdata.508946 2026-02-13T02:29:54.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj326 testdata.508946 2026-02-13T02:29:54.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj327 testdata.508946 2026-02-13T02:29:54.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj328 testdata.508946 2026-02-13T02:29:54.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:54.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj329 testdata.508946 2026-02-13T02:29:55.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.015 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj330 testdata.508946 2026-02-13T02:29:55.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj331 testdata.508946 2026-02-13T02:29:55.064 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.064 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj332 testdata.508946 2026-02-13T02:29:55.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj333 testdata.508946 2026-02-13T02:29:55.108 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.108 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj334 testdata.508946 2026-02-13T02:29:55.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj335 testdata.508946 2026-02-13T02:29:55.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj336 testdata.508946 2026-02-13T02:29:55.170 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.170 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj337 testdata.508946 2026-02-13T02:29:55.195 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.195 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj338 testdata.508946 2026-02-13T02:29:55.219 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.219 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj339 testdata.508946 2026-02-13T02:29:55.243 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.243 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj340 testdata.508946 2026-02-13T02:29:55.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj341 testdata.508946 2026-02-13T02:29:55.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj342 testdata.508946 2026-02-13T02:29:55.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj343 testdata.508946 2026-02-13T02:29:55.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj344 testdata.508946 2026-02-13T02:29:55.361 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.361 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj345 testdata.508946 2026-02-13T02:29:55.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj346 testdata.508946 2026-02-13T02:29:55.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj347 testdata.508946 2026-02-13T02:29:55.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj348 testdata.508946 2026-02-13T02:29:55.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj349 testdata.508946 2026-02-13T02:29:55.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj350 testdata.508946 2026-02-13T02:29:55.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj351 testdata.508946 2026-02-13T02:29:55.527 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj352 testdata.508946 2026-02-13T02:29:55.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj353 testdata.508946 2026-02-13T02:29:55.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj354 testdata.508946 2026-02-13T02:29:55.601 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.601 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj355 testdata.508946 2026-02-13T02:29:55.626 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.626 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj356 testdata.508946 2026-02-13T02:29:55.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj357 testdata.508946 2026-02-13T02:29:55.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj358 testdata.508946 2026-02-13T02:29:55.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj359 testdata.508946 2026-02-13T02:29:55.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj360 testdata.508946 2026-02-13T02:29:55.739 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.739 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj361 testdata.508946 2026-02-13T02:29:55.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj362 testdata.508946 2026-02-13T02:29:55.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj363 testdata.508946 2026-02-13T02:29:55.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.801 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj364 testdata.508946 2026-02-13T02:29:55.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj365 testdata.508946 2026-02-13T02:29:55.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj366 testdata.508946 2026-02-13T02:29:55.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj367 testdata.508946 2026-02-13T02:29:55.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj368 testdata.508946 2026-02-13T02:29:55.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.910 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj369 testdata.508946 2026-02-13T02:29:55.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj370 testdata.508946 2026-02-13T02:29:55.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj371 testdata.508946 2026-02-13T02:29:55.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:55.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj372 testdata.508946 2026-02-13T02:29:56.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj373 testdata.508946 2026-02-13T02:29:56.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj374 testdata.508946 2026-02-13T02:29:56.055 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.055 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj375 testdata.508946 2026-02-13T02:29:56.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj376 testdata.508946 2026-02-13T02:29:56.104 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.104 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj377 testdata.508946 2026-02-13T02:29:56.129 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.129 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj378 testdata.508946 2026-02-13T02:29:56.154 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.154 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj379 testdata.508946 2026-02-13T02:29:56.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj380 testdata.508946 2026-02-13T02:29:56.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj381 testdata.508946 2026-02-13T02:29:56.229 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.229 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj382 testdata.508946 2026-02-13T02:29:56.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj383 testdata.508946 2026-02-13T02:29:56.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj384 testdata.508946 2026-02-13T02:29:56.291 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.291 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj385 testdata.508946 2026-02-13T02:29:56.315 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj386 testdata.508946 2026-02-13T02:29:56.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj387 testdata.508946 2026-02-13T02:29:56.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj388 testdata.508946 2026-02-13T02:29:56.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj389 testdata.508946 2026-02-13T02:29:56.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj390 testdata.508946 2026-02-13T02:29:56.439 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj391 testdata.508946 2026-02-13T02:29:56.465 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.465 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj392 testdata.508946 2026-02-13T02:29:56.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj393 testdata.508946 2026-02-13T02:29:56.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj394 testdata.508946 2026-02-13T02:29:56.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj395 testdata.508946 2026-02-13T02:29:56.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj396 testdata.508946 2026-02-13T02:29:56.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj397 testdata.508946 2026-02-13T02:29:56.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj398 testdata.508946 2026-02-13T02:29:56.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj399 testdata.508946 2026-02-13T02:29:56.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj400 testdata.508946 2026-02-13T02:29:56.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj401 testdata.508946 2026-02-13T02:29:56.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj402 testdata.508946 2026-02-13T02:29:56.739 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.739 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj403 testdata.508946 2026-02-13T02:29:56.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj404 testdata.508946 2026-02-13T02:29:56.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj405 testdata.508946 2026-02-13T02:29:56.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj406 testdata.508946 2026-02-13T02:29:56.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj407 testdata.508946 2026-02-13T02:29:56.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj408 testdata.508946 2026-02-13T02:29:56.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj409 testdata.508946 2026-02-13T02:29:56.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj410 testdata.508946 2026-02-13T02:29:56.925 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.925 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj411 testdata.508946 2026-02-13T02:29:56.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj412 testdata.508946 2026-02-13T02:29:56.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj413 testdata.508946 2026-02-13T02:29:56.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:56.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj414 testdata.508946 2026-02-13T02:29:57.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj415 testdata.508946 2026-02-13T02:29:57.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj416 testdata.508946 2026-02-13T02:29:57.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj417 testdata.508946 2026-02-13T02:29:57.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj418 testdata.508946 2026-02-13T02:29:57.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj419 testdata.508946 2026-02-13T02:29:57.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj420 testdata.508946 2026-02-13T02:29:57.154 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.154 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj421 testdata.508946 2026-02-13T02:29:57.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj422 testdata.508946 2026-02-13T02:29:57.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj423 testdata.508946 2026-02-13T02:29:57.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj424 testdata.508946 2026-02-13T02:29:57.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj425 testdata.508946 2026-02-13T02:29:57.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj426 testdata.508946 2026-02-13T02:29:57.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj427 testdata.508946 2026-02-13T02:29:57.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj428 testdata.508946 2026-02-13T02:29:57.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj429 testdata.508946 2026-02-13T02:29:57.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj430 testdata.508946 2026-02-13T02:29:57.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj431 testdata.508946 2026-02-13T02:29:57.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj432 testdata.508946 2026-02-13T02:29:57.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj433 testdata.508946 2026-02-13T02:29:57.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj434 testdata.508946 2026-02-13T02:29:57.469 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.469 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj435 testdata.508946 2026-02-13T02:29:57.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj436 testdata.508946 2026-02-13T02:29:57.516 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.516 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj437 testdata.508946 2026-02-13T02:29:57.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj438 testdata.508946 2026-02-13T02:29:57.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj439 testdata.508946 2026-02-13T02:29:57.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj440 testdata.508946 2026-02-13T02:29:57.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj441 testdata.508946 2026-02-13T02:29:57.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj442 testdata.508946 2026-02-13T02:29:57.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj443 testdata.508946 2026-02-13T02:29:57.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj444 testdata.508946 2026-02-13T02:29:57.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj445 testdata.508946 2026-02-13T02:29:57.737 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.737 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj446 testdata.508946 2026-02-13T02:29:57.762 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj447 testdata.508946 2026-02-13T02:29:57.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj448 testdata.508946 2026-02-13T02:29:57.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj449 testdata.508946 2026-02-13T02:29:57.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj450 testdata.508946 2026-02-13T02:29:57.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj451 testdata.508946 2026-02-13T02:29:57.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj452 testdata.508946 2026-02-13T02:29:57.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj453 testdata.508946 2026-02-13T02:29:57.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj454 testdata.508946 2026-02-13T02:29:57.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj455 testdata.508946 2026-02-13T02:29:57.970 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.970 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj456 testdata.508946 2026-02-13T02:29:57.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:57.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj457 testdata.508946 2026-02-13T02:29:58.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj458 testdata.508946 2026-02-13T02:29:58.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj459 testdata.508946 2026-02-13T02:29:58.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj460 testdata.508946 2026-02-13T02:29:58.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj461 testdata.508946 2026-02-13T02:29:58.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj462 testdata.508946 2026-02-13T02:29:58.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj463 testdata.508946 2026-02-13T02:29:58.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj464 testdata.508946 2026-02-13T02:29:58.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj465 testdata.508946 2026-02-13T02:29:58.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj466 testdata.508946 2026-02-13T02:29:58.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj467 testdata.508946 2026-02-13T02:29:58.254 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.255 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj468 testdata.508946 2026-02-13T02:29:58.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj469 testdata.508946 2026-02-13T02:29:58.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj470 testdata.508946 2026-02-13T02:29:58.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj471 testdata.508946 2026-02-13T02:29:58.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj472 testdata.508946 2026-02-13T02:29:58.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj473 testdata.508946 2026-02-13T02:29:58.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj474 testdata.508946 2026-02-13T02:29:58.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj475 testdata.508946 2026-02-13T02:29:58.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj476 testdata.508946 2026-02-13T02:29:58.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj477 testdata.508946 2026-02-13T02:29:58.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj478 testdata.508946 2026-02-13T02:29:58.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj479 testdata.508946 2026-02-13T02:29:58.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj480 testdata.508946 2026-02-13T02:29:58.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj481 testdata.508946 2026-02-13T02:29:58.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj482 testdata.508946 2026-02-13T02:29:58.621 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.621 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj483 testdata.508946 2026-02-13T02:29:58.645 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.646 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj484 testdata.508946 2026-02-13T02:29:58.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj485 testdata.508946 2026-02-13T02:29:58.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj486 testdata.508946 2026-02-13T02:29:58.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj487 testdata.508946 2026-02-13T02:29:58.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj488 testdata.508946 2026-02-13T02:29:58.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj489 testdata.508946 2026-02-13T02:29:58.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj490 testdata.508946 2026-02-13T02:29:58.799 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.799 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj491 testdata.508946 2026-02-13T02:29:58.816 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.816 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj492 testdata.508946 2026-02-13T02:29:58.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj493 testdata.508946 2026-02-13T02:29:58.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj494 testdata.508946 2026-02-13T02:29:58.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj495 testdata.508946 2026-02-13T02:29:58.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj496 testdata.508946 2026-02-13T02:29:58.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj497 testdata.508946 2026-02-13T02:29:58.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj498 testdata.508946 2026-02-13T02:29:58.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:58.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj499 testdata.508946 2026-02-13T02:29:59.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj500 testdata.508946 2026-02-13T02:29:59.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj501 testdata.508946 2026-02-13T02:29:59.051 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.051 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj502 testdata.508946 2026-02-13T02:29:59.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj503 testdata.508946 2026-02-13T02:29:59.099 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.099 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj504 testdata.508946 2026-02-13T02:29:59.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj505 testdata.508946 2026-02-13T02:29:59.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj506 testdata.508946 2026-02-13T02:29:59.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj507 testdata.508946 2026-02-13T02:29:59.198 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.198 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj508 testdata.508946 2026-02-13T02:29:59.222 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.223 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj509 testdata.508946 2026-02-13T02:29:59.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj510 testdata.508946 2026-02-13T02:29:59.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj511 testdata.508946 2026-02-13T02:29:59.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj512 testdata.508946 2026-02-13T02:29:59.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj513 testdata.508946 2026-02-13T02:29:59.339 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.339 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj514 testdata.508946 2026-02-13T02:29:59.358 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.358 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj515 testdata.508946 2026-02-13T02:29:59.381 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.381 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj516 testdata.508946 2026-02-13T02:29:59.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj517 testdata.508946 2026-02-13T02:29:59.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj518 testdata.508946 2026-02-13T02:29:59.456 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.456 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj519 testdata.508946 2026-02-13T02:29:59.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj520 testdata.508946 2026-02-13T02:29:59.505 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.505 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj521 testdata.508946 2026-02-13T02:29:59.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj522 testdata.508946 2026-02-13T02:29:59.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj523 testdata.508946 2026-02-13T02:29:59.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj524 testdata.508946 2026-02-13T02:29:59.597 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.598 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj525 testdata.508946 2026-02-13T02:29:59.621 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.621 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj526 testdata.508946 2026-02-13T02:29:59.645 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.645 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj527 testdata.508946 2026-02-13T02:29:59.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj528 testdata.508946 2026-02-13T02:29:59.694 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.694 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj529 testdata.508946 2026-02-13T02:29:59.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.718 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj530 testdata.508946 2026-02-13T02:29:59.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj531 testdata.508946 2026-02-13T02:29:59.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj532 testdata.508946 2026-02-13T02:29:59.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj533 testdata.508946 2026-02-13T02:29:59.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj534 testdata.508946 2026-02-13T02:29:59.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj535 testdata.508946 2026-02-13T02:29:59.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj536 testdata.508946 2026-02-13T02:29:59.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj537 testdata.508946 2026-02-13T02:29:59.892 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.892 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj538 testdata.508946 2026-02-13T02:29:59.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj539 testdata.508946 2026-02-13T02:29:59.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj540 testdata.508946 2026-02-13T02:29:59.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj541 testdata.508946 2026-02-13T02:29:59.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:29:59.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj542 testdata.508946 2026-02-13T02:30:00.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj543 testdata.508946 2026-02-13T02:30:00.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj544 testdata.508946 2026-02-13T02:30:00.047 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.047 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj545 testdata.508946 2026-02-13T02:30:00.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj546 testdata.508946 2026-02-13T02:30:00.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj547 testdata.508946 2026-02-13T02:30:00.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj548 testdata.508946 2026-02-13T02:30:00.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj549 testdata.508946 2026-02-13T02:30:00.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj550 testdata.508946 2026-02-13T02:30:00.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj551 testdata.508946 2026-02-13T02:30:00.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj552 testdata.508946 2026-02-13T02:30:00.181 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.181 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj553 testdata.508946 2026-02-13T02:30:00.198 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.198 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj554 testdata.508946 2026-02-13T02:30:00.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj555 testdata.508946 2026-02-13T02:30:00.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj556 testdata.508946 2026-02-13T02:30:00.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj557 testdata.508946 2026-02-13T02:30:00.267 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.267 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj558 testdata.508946 2026-02-13T02:30:00.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj559 testdata.508946 2026-02-13T02:30:00.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj560 testdata.508946 2026-02-13T02:30:00.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj561 testdata.508946 2026-02-13T02:30:00.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj562 testdata.508946 2026-02-13T02:30:00.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj563 testdata.508946 2026-02-13T02:30:00.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj564 testdata.508946 2026-02-13T02:30:00.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj565 testdata.508946 2026-02-13T02:30:00.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.414 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj566 testdata.508946 2026-02-13T02:30:00.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj567 testdata.508946 2026-02-13T02:30:00.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj568 testdata.508946 2026-02-13T02:30:00.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj569 testdata.508946 2026-02-13T02:30:00.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj570 testdata.508946 2026-02-13T02:30:00.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj571 testdata.508946 2026-02-13T02:30:00.516 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.516 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj572 testdata.508946 2026-02-13T02:30:00.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj573 testdata.508946 2026-02-13T02:30:00.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj574 testdata.508946 2026-02-13T02:30:00.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj575 testdata.508946 2026-02-13T02:30:00.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj576 testdata.508946 2026-02-13T02:30:00.618 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.618 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj577 testdata.508946 2026-02-13T02:30:00.643 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.643 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj578 testdata.508946 2026-02-13T02:30:00.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj579 testdata.508946 2026-02-13T02:30:00.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj580 testdata.508946 2026-02-13T02:30:00.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj581 testdata.508946 2026-02-13T02:30:00.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj582 testdata.508946 2026-02-13T02:30:00.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj583 testdata.508946 2026-02-13T02:30:00.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj584 testdata.508946 2026-02-13T02:30:00.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj585 testdata.508946 2026-02-13T02:30:00.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj586 testdata.508946 2026-02-13T02:30:00.847 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.847 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj587 testdata.508946 2026-02-13T02:30:00.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj588 testdata.508946 2026-02-13T02:30:00.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj589 testdata.508946 2026-02-13T02:30:00.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj590 testdata.508946 2026-02-13T02:30:00.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj591 testdata.508946 2026-02-13T02:30:00.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj592 testdata.508946 2026-02-13T02:30:00.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj593 testdata.508946 2026-02-13T02:30:00.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj594 testdata.508946 2026-02-13T02:30:00.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:00.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj595 testdata.508946 2026-02-13T02:30:01.016 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.016 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj596 testdata.508946 2026-02-13T02:30:01.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj597 testdata.508946 2026-02-13T02:30:01.057 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.057 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj598 testdata.508946 2026-02-13T02:30:01.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj599 testdata.508946 2026-02-13T02:30:01.105 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.105 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj600 testdata.508946 2026-02-13T02:30:01.129 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj601 testdata.508946 2026-02-13T02:30:01.154 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.154 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj602 testdata.508946 2026-02-13T02:30:01.178 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.178 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj603 testdata.508946 2026-02-13T02:30:01.203 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.203 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj604 testdata.508946 2026-02-13T02:30:01.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj605 testdata.508946 2026-02-13T02:30:01.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj606 testdata.508946 2026-02-13T02:30:01.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj607 testdata.508946 2026-02-13T02:30:01.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj608 testdata.508946 2026-02-13T02:30:01.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj609 testdata.508946 2026-02-13T02:30:01.339 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.339 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj610 testdata.508946 2026-02-13T02:30:01.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj611 testdata.508946 2026-02-13T02:30:01.385 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.385 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj612 testdata.508946 2026-02-13T02:30:01.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj613 testdata.508946 2026-02-13T02:30:01.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj614 testdata.508946 2026-02-13T02:30:01.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj615 testdata.508946 2026-02-13T02:30:01.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj616 testdata.508946 2026-02-13T02:30:01.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj617 testdata.508946 2026-02-13T02:30:01.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj618 testdata.508946 2026-02-13T02:30:01.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj619 testdata.508946 2026-02-13T02:30:01.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj620 testdata.508946 2026-02-13T02:30:01.594 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.594 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj621 testdata.508946 2026-02-13T02:30:01.619 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.619 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj622 testdata.508946 2026-02-13T02:30:01.643 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.644 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj623 testdata.508946 2026-02-13T02:30:01.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj624 testdata.508946 2026-02-13T02:30:01.692 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.692 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj625 testdata.508946 2026-02-13T02:30:01.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj626 testdata.508946 2026-02-13T02:30:01.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj627 testdata.508946 2026-02-13T02:30:01.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj628 testdata.508946 2026-02-13T02:30:01.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj629 testdata.508946 2026-02-13T02:30:01.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj630 testdata.508946 2026-02-13T02:30:01.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj631 testdata.508946 2026-02-13T02:30:01.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj632 testdata.508946 2026-02-13T02:30:01.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj633 testdata.508946 2026-02-13T02:30:01.904 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.904 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj634 testdata.508946 2026-02-13T02:30:01.929 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.929 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj635 testdata.508946 2026-02-13T02:30:01.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj636 testdata.508946 2026-02-13T02:30:01.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:01.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj637 testdata.508946 2026-02-13T02:30:02.001 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj638 testdata.508946 2026-02-13T02:30:02.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj639 testdata.508946 2026-02-13T02:30:02.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj640 testdata.508946 2026-02-13T02:30:02.056 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.056 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj641 testdata.508946 2026-02-13T02:30:02.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj642 testdata.508946 2026-02-13T02:30:02.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj643 testdata.508946 2026-02-13T02:30:02.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj644 testdata.508946 2026-02-13T02:30:02.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj645 testdata.508946 2026-02-13T02:30:02.176 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.176 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj646 testdata.508946 2026-02-13T02:30:02.201 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.201 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj647 testdata.508946 2026-02-13T02:30:02.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj648 testdata.508946 2026-02-13T02:30:02.251 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.251 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj649 testdata.508946 2026-02-13T02:30:02.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj650 testdata.508946 2026-02-13T02:30:02.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj651 testdata.508946 2026-02-13T02:30:02.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj652 testdata.508946 2026-02-13T02:30:02.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj653 testdata.508946 2026-02-13T02:30:02.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj654 testdata.508946 2026-02-13T02:30:02.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj655 testdata.508946 2026-02-13T02:30:02.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj656 testdata.508946 2026-02-13T02:30:02.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj657 testdata.508946 2026-02-13T02:30:02.474 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.474 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj658 testdata.508946 2026-02-13T02:30:02.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj659 testdata.508946 2026-02-13T02:30:02.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj660 testdata.508946 2026-02-13T02:30:02.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj661 testdata.508946 2026-02-13T02:30:02.574 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.574 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj662 testdata.508946 2026-02-13T02:30:02.598 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.598 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj663 testdata.508946 2026-02-13T02:30:02.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj664 testdata.508946 2026-02-13T02:30:02.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj665 testdata.508946 2026-02-13T02:30:02.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj666 testdata.508946 2026-02-13T02:30:02.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj667 testdata.508946 2026-02-13T02:30:02.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj668 testdata.508946 2026-02-13T02:30:02.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj669 testdata.508946 2026-02-13T02:30:02.770 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.770 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj670 testdata.508946 2026-02-13T02:30:02.795 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.795 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj671 testdata.508946 2026-02-13T02:30:02.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj672 testdata.508946 2026-02-13T02:30:02.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj673 testdata.508946 2026-02-13T02:30:02.871 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.871 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj674 testdata.508946 2026-02-13T02:30:02.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj675 testdata.508946 2026-02-13T02:30:02.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj676 testdata.508946 2026-02-13T02:30:02.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj677 testdata.508946 2026-02-13T02:30:02.971 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.971 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj678 testdata.508946 2026-02-13T02:30:02.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:02.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj679 testdata.508946 2026-02-13T02:30:03.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj680 testdata.508946 2026-02-13T02:30:03.037 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.037 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj681 testdata.508946 2026-02-13T02:30:03.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj682 testdata.508946 2026-02-13T02:30:03.086 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj683 testdata.508946 2026-02-13T02:30:03.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj684 testdata.508946 2026-02-13T02:30:03.136 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj685 testdata.508946 2026-02-13T02:30:03.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj686 testdata.508946 2026-02-13T02:30:03.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj687 testdata.508946 2026-02-13T02:30:03.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj688 testdata.508946 2026-02-13T02:30:03.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj689 testdata.508946 2026-02-13T02:30:03.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj690 testdata.508946 2026-02-13T02:30:03.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj691 testdata.508946 2026-02-13T02:30:03.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj692 testdata.508946 2026-02-13T02:30:03.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj693 testdata.508946 2026-02-13T02:30:03.359 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.359 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj694 testdata.508946 2026-02-13T02:30:03.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj695 testdata.508946 2026-02-13T02:30:03.401 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.402 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj696 testdata.508946 2026-02-13T02:30:03.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj697 testdata.508946 2026-02-13T02:30:03.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj698 testdata.508946 2026-02-13T02:30:03.465 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.465 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj699 testdata.508946 2026-02-13T02:30:03.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj700 testdata.508946 2026-02-13T02:30:03.509 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.510 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj701 testdata.508946 2026-02-13T02:30:03.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj702 testdata.508946 2026-02-13T02:30:03.557 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.557 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj703 testdata.508946 2026-02-13T02:30:03.575 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.575 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj704 testdata.508946 2026-02-13T02:30:03.597 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.597 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj705 testdata.508946 2026-02-13T02:30:03.621 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.621 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj706 testdata.508946 2026-02-13T02:30:03.645 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.646 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj707 testdata.508946 2026-02-13T02:30:03.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj708 testdata.508946 2026-02-13T02:30:03.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj709 testdata.508946 2026-02-13T02:30:03.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj710 testdata.508946 2026-02-13T02:30:03.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj711 testdata.508946 2026-02-13T02:30:03.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj712 testdata.508946 2026-02-13T02:30:03.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj713 testdata.508946 2026-02-13T02:30:03.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj714 testdata.508946 2026-02-13T02:30:03.839 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.839 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj715 testdata.508946 2026-02-13T02:30:03.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj716 testdata.508946 2026-02-13T02:30:03.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj717 testdata.508946 2026-02-13T02:30:03.914 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.914 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj718 testdata.508946 2026-02-13T02:30:03.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj719 testdata.508946 2026-02-13T02:30:03.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj720 testdata.508946 2026-02-13T02:30:03.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:03.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj721 testdata.508946 2026-02-13T02:30:04.006 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj722 testdata.508946 2026-02-13T02:30:04.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj723 testdata.508946 2026-02-13T02:30:04.053 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.053 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj724 testdata.508946 2026-02-13T02:30:04.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj725 testdata.508946 2026-02-13T02:30:04.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj726 testdata.508946 2026-02-13T02:30:04.122 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.123 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj727 testdata.508946 2026-02-13T02:30:04.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj728 testdata.508946 2026-02-13T02:30:04.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj729 testdata.508946 2026-02-13T02:30:04.196 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.196 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj730 testdata.508946 2026-02-13T02:30:04.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj731 testdata.508946 2026-02-13T02:30:04.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj732 testdata.508946 2026-02-13T02:30:04.271 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.271 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj733 testdata.508946 2026-02-13T02:30:04.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj734 testdata.508946 2026-02-13T02:30:04.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj735 testdata.508946 2026-02-13T02:30:04.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj736 testdata.508946 2026-02-13T02:30:04.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj737 testdata.508946 2026-02-13T02:30:04.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj738 testdata.508946 2026-02-13T02:30:04.393 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.394 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj739 testdata.508946 2026-02-13T02:30:04.416 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.416 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj740 testdata.508946 2026-02-13T02:30:04.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj741 testdata.508946 2026-02-13T02:30:04.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj742 testdata.508946 2026-02-13T02:30:04.465 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj743 testdata.508946 2026-02-13T02:30:04.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj744 testdata.508946 2026-02-13T02:30:04.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj745 testdata.508946 2026-02-13T02:30:04.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj746 testdata.508946 2026-02-13T02:30:04.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj747 testdata.508946 2026-02-13T02:30:04.568 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.568 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj748 testdata.508946 2026-02-13T02:30:04.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj749 testdata.508946 2026-02-13T02:30:04.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj750 testdata.508946 2026-02-13T02:30:04.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj751 testdata.508946 2026-02-13T02:30:04.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj752 testdata.508946 2026-02-13T02:30:04.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj753 testdata.508946 2026-02-13T02:30:04.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj754 testdata.508946 2026-02-13T02:30:04.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.735 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj755 testdata.508946 2026-02-13T02:30:04.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj756 testdata.508946 2026-02-13T02:30:04.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj757 testdata.508946 2026-02-13T02:30:04.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj758 testdata.508946 2026-02-13T02:30:04.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj759 testdata.508946 2026-02-13T02:30:04.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj760 testdata.508946 2026-02-13T02:30:04.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj761 testdata.508946 2026-02-13T02:30:04.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.908 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj762 testdata.508946 2026-02-13T02:30:04.925 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.926 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj763 testdata.508946 2026-02-13T02:30:04.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.950 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj764 testdata.508946 2026-02-13T02:30:04.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj765 testdata.508946 2026-02-13T02:30:04.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:04.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj766 testdata.508946 2026-02-13T02:30:05.015 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.015 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj767 testdata.508946 2026-02-13T02:30:05.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj768 testdata.508946 2026-02-13T02:30:05.064 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.064 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj769 testdata.508946 2026-02-13T02:30:05.089 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.089 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj770 testdata.508946 2026-02-13T02:30:05.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj771 testdata.508946 2026-02-13T02:30:05.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj772 testdata.508946 2026-02-13T02:30:05.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj773 testdata.508946 2026-02-13T02:30:05.183 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.183 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj774 testdata.508946 2026-02-13T02:30:05.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj775 testdata.508946 2026-02-13T02:30:05.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj776 testdata.508946 2026-02-13T02:30:05.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj777 testdata.508946 2026-02-13T02:30:05.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj778 testdata.508946 2026-02-13T02:30:05.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj779 testdata.508946 2026-02-13T02:30:05.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj780 testdata.508946 2026-02-13T02:30:05.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj781 testdata.508946 2026-02-13T02:30:05.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj782 testdata.508946 2026-02-13T02:30:05.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj783 testdata.508946 2026-02-13T02:30:05.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj784 testdata.508946 2026-02-13T02:30:05.439 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj785 testdata.508946 2026-02-13T02:30:05.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj786 testdata.508946 2026-02-13T02:30:05.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj787 testdata.508946 2026-02-13T02:30:05.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj788 testdata.508946 2026-02-13T02:30:05.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj789 testdata.508946 2026-02-13T02:30:05.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj790 testdata.508946 2026-02-13T02:30:05.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj791 testdata.508946 2026-02-13T02:30:05.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj792 testdata.508946 2026-02-13T02:30:05.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj793 testdata.508946 2026-02-13T02:30:05.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj794 testdata.508946 2026-02-13T02:30:05.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj795 testdata.508946 2026-02-13T02:30:05.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj796 testdata.508946 2026-02-13T02:30:05.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj797 testdata.508946 2026-02-13T02:30:05.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj798 testdata.508946 2026-02-13T02:30:05.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.766 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj799 testdata.508946 2026-02-13T02:30:05.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj800 testdata.508946 2026-02-13T02:30:05.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj801 testdata.508946 2026-02-13T02:30:05.839 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.839 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj802 testdata.508946 2026-02-13T02:30:05.857 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj803 testdata.508946 2026-02-13T02:30:05.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj804 testdata.508946 2026-02-13T02:30:05.906 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.906 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj805 testdata.508946 2026-02-13T02:30:05.924 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.924 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj806 testdata.508946 2026-02-13T02:30:05.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj807 testdata.508946 2026-02-13T02:30:05.970 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.970 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj808 testdata.508946 2026-02-13T02:30:05.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:05.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj809 testdata.508946 2026-02-13T02:30:06.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj810 testdata.508946 2026-02-13T02:30:06.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj811 testdata.508946 2026-02-13T02:30:06.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj812 testdata.508946 2026-02-13T02:30:06.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj813 testdata.508946 2026-02-13T02:30:06.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj814 testdata.508946 2026-02-13T02:30:06.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj815 testdata.508946 2026-02-13T02:30:06.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj816 testdata.508946 2026-02-13T02:30:06.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj817 testdata.508946 2026-02-13T02:30:06.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj818 testdata.508946 2026-02-13T02:30:06.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj819 testdata.508946 2026-02-13T02:30:06.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj820 testdata.508946 2026-02-13T02:30:06.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj821 testdata.508946 2026-02-13T02:30:06.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj822 testdata.508946 2026-02-13T02:30:06.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj823 testdata.508946 2026-02-13T02:30:06.347 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.347 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj824 testdata.508946 2026-02-13T02:30:06.372 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.372 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj825 testdata.508946 2026-02-13T02:30:06.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj826 testdata.508946 2026-02-13T02:30:06.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj827 testdata.508946 2026-02-13T02:30:06.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj828 testdata.508946 2026-02-13T02:30:06.463 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.463 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj829 testdata.508946 2026-02-13T02:30:06.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj830 testdata.508946 2026-02-13T02:30:06.506 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.506 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj831 testdata.508946 2026-02-13T02:30:06.523 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.523 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj832 testdata.508946 2026-02-13T02:30:06.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj833 testdata.508946 2026-02-13T02:30:06.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj834 testdata.508946 2026-02-13T02:30:06.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj835 testdata.508946 2026-02-13T02:30:06.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj836 testdata.508946 2026-02-13T02:30:06.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj837 testdata.508946 2026-02-13T02:30:06.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj838 testdata.508946 2026-02-13T02:30:06.678 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.678 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj839 testdata.508946 2026-02-13T02:30:06.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj840 testdata.508946 2026-02-13T02:30:06.718 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj841 testdata.508946 2026-02-13T02:30:06.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj842 testdata.508946 2026-02-13T02:30:06.767 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.767 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj843 testdata.508946 2026-02-13T02:30:06.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj844 testdata.508946 2026-02-13T02:30:06.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.806 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj845 testdata.508946 2026-02-13T02:30:06.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj846 testdata.508946 2026-02-13T02:30:06.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj847 testdata.508946 2026-02-13T02:30:06.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj848 testdata.508946 2026-02-13T02:30:06.904 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.905 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj849 testdata.508946 2026-02-13T02:30:06.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj850 testdata.508946 2026-02-13T02:30:06.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj851 testdata.508946 2026-02-13T02:30:06.980 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.980 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj852 testdata.508946 2026-02-13T02:30:06.998 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:06.998 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj853 testdata.508946 2026-02-13T02:30:07.023 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.023 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj854 testdata.508946 2026-02-13T02:30:07.047 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.047 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj855 testdata.508946 2026-02-13T02:30:07.064 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.064 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj856 testdata.508946 2026-02-13T02:30:07.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj857 testdata.508946 2026-02-13T02:30:07.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj858 testdata.508946 2026-02-13T02:30:07.136 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.136 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj859 testdata.508946 2026-02-13T02:30:07.160 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.160 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj860 testdata.508946 2026-02-13T02:30:07.180 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.180 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj861 testdata.508946 2026-02-13T02:30:07.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj862 testdata.508946 2026-02-13T02:30:07.230 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.230 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj863 testdata.508946 2026-02-13T02:30:07.254 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.255 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj864 testdata.508946 2026-02-13T02:30:07.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj865 testdata.508946 2026-02-13T02:30:07.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj866 testdata.508946 2026-02-13T02:30:07.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj867 testdata.508946 2026-02-13T02:30:07.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj868 testdata.508946 2026-02-13T02:30:07.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj869 testdata.508946 2026-02-13T02:30:07.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj870 testdata.508946 2026-02-13T02:30:07.410 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.410 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj871 testdata.508946 2026-02-13T02:30:07.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj872 testdata.508946 2026-02-13T02:30:07.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj873 testdata.508946 2026-02-13T02:30:07.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj874 testdata.508946 2026-02-13T02:30:07.506 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.506 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj875 testdata.508946 2026-02-13T02:30:07.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj876 testdata.508946 2026-02-13T02:30:07.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj877 testdata.508946 2026-02-13T02:30:07.574 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.574 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj878 testdata.508946 2026-02-13T02:30:07.598 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.598 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj879 testdata.508946 2026-02-13T02:30:07.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj880 testdata.508946 2026-02-13T02:30:07.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj881 testdata.508946 2026-02-13T02:30:07.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj882 testdata.508946 2026-02-13T02:30:07.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj883 testdata.508946 2026-02-13T02:30:07.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj884 testdata.508946 2026-02-13T02:30:07.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj885 testdata.508946 2026-02-13T02:30:07.771 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.771 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj886 testdata.508946 2026-02-13T02:30:07.795 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj887 testdata.508946 2026-02-13T02:30:07.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj888 testdata.508946 2026-02-13T02:30:07.845 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.845 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj889 testdata.508946 2026-02-13T02:30:07.869 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.869 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj890 testdata.508946 2026-02-13T02:30:07.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj891 testdata.508946 2026-02-13T02:30:07.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj892 testdata.508946 2026-02-13T02:30:07.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj893 testdata.508946 2026-02-13T02:30:07.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj894 testdata.508946 2026-02-13T02:30:07.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:07.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj895 testdata.508946 2026-02-13T02:30:08.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj896 testdata.508946 2026-02-13T02:30:08.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj897 testdata.508946 2026-02-13T02:30:08.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj898 testdata.508946 2026-02-13T02:30:08.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj899 testdata.508946 2026-02-13T02:30:08.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj900 testdata.508946 2026-02-13T02:30:08.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj901 testdata.508946 2026-02-13T02:30:08.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj902 testdata.508946 2026-02-13T02:30:08.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj903 testdata.508946 2026-02-13T02:30:08.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj904 testdata.508946 2026-02-13T02:30:08.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj905 testdata.508946 2026-02-13T02:30:08.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj906 testdata.508946 2026-02-13T02:30:08.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj907 testdata.508946 2026-02-13T02:30:08.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj908 testdata.508946 2026-02-13T02:30:08.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj909 testdata.508946 2026-02-13T02:30:08.357 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.357 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj910 testdata.508946 2026-02-13T02:30:08.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj911 testdata.508946 2026-02-13T02:30:08.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj912 testdata.508946 2026-02-13T02:30:08.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj913 testdata.508946 2026-02-13T02:30:08.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj914 testdata.508946 2026-02-13T02:30:08.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj915 testdata.508946 2026-02-13T02:30:08.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj916 testdata.508946 2026-02-13T02:30:08.527 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.527 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj917 testdata.508946 2026-02-13T02:30:08.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj918 testdata.508946 2026-02-13T02:30:08.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj919 testdata.508946 2026-02-13T02:30:08.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj920 testdata.508946 2026-02-13T02:30:08.627 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.627 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj921 testdata.508946 2026-02-13T02:30:08.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj922 testdata.508946 2026-02-13T02:30:08.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj923 testdata.508946 2026-02-13T02:30:08.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj924 testdata.508946 2026-02-13T02:30:08.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj925 testdata.508946 2026-02-13T02:30:08.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj926 testdata.508946 2026-02-13T02:30:08.775 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.775 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj927 testdata.508946 2026-02-13T02:30:08.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj928 testdata.508946 2026-02-13T02:30:08.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj929 testdata.508946 2026-02-13T02:30:08.836 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.836 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj930 testdata.508946 2026-02-13T02:30:08.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj931 testdata.508946 2026-02-13T02:30:08.877 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.878 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj932 testdata.508946 2026-02-13T02:30:08.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj933 testdata.508946 2026-02-13T02:30:08.926 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.926 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj934 testdata.508946 2026-02-13T02:30:08.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj935 testdata.508946 2026-02-13T02:30:08.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj936 testdata.508946 2026-02-13T02:30:08.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:08.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj937 testdata.508946 2026-02-13T02:30:09.017 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj938 testdata.508946 2026-02-13T02:30:09.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.042 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj939 testdata.508946 2026-02-13T02:30:09.065 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.065 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj940 testdata.508946 2026-02-13T02:30:09.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj941 testdata.508946 2026-02-13T02:30:09.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj942 testdata.508946 2026-02-13T02:30:09.123 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.123 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj943 testdata.508946 2026-02-13T02:30:09.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj944 testdata.508946 2026-02-13T02:30:09.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj945 testdata.508946 2026-02-13T02:30:09.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj946 testdata.508946 2026-02-13T02:30:09.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj947 testdata.508946 2026-02-13T02:30:09.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj948 testdata.508946 2026-02-13T02:30:09.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj949 testdata.508946 2026-02-13T02:30:09.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj950 testdata.508946 2026-02-13T02:30:09.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj951 testdata.508946 2026-02-13T02:30:09.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj952 testdata.508946 2026-02-13T02:30:09.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj953 testdata.508946 2026-02-13T02:30:09.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj954 testdata.508946 2026-02-13T02:30:09.403 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj955 testdata.508946 2026-02-13T02:30:09.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj956 testdata.508946 2026-02-13T02:30:09.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj957 testdata.508946 2026-02-13T02:30:09.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj958 testdata.508946 2026-02-13T02:30:09.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj959 testdata.508946 2026-02-13T02:30:09.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj960 testdata.508946 2026-02-13T02:30:09.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj961 testdata.508946 2026-02-13T02:30:09.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj962 testdata.508946 2026-02-13T02:30:09.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj963 testdata.508946 2026-02-13T02:30:09.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj964 testdata.508946 2026-02-13T02:30:09.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj965 testdata.508946 2026-02-13T02:30:09.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj966 testdata.508946 2026-02-13T02:30:09.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj967 testdata.508946 2026-02-13T02:30:09.718 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.718 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj968 testdata.508946 2026-02-13T02:30:09.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj969 testdata.508946 2026-02-13T02:30:09.767 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.767 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj970 testdata.508946 2026-02-13T02:30:09.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj971 testdata.508946 2026-02-13T02:30:09.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj972 testdata.508946 2026-02-13T02:30:09.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj973 testdata.508946 2026-02-13T02:30:09.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj974 testdata.508946 2026-02-13T02:30:09.869 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.869 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj975 testdata.508946 2026-02-13T02:30:09.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.892 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj976 testdata.508946 2026-02-13T02:30:09.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj977 testdata.508946 2026-02-13T02:30:09.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj978 testdata.508946 2026-02-13T02:30:09.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj979 testdata.508946 2026-02-13T02:30:09.980 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:09.980 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj980 testdata.508946 2026-02-13T02:30:10.004 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:10.004 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj981 testdata.508946 2026-02-13T02:30:10.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:10.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj982 testdata.508946 2026-02-13T02:30:10.050 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:10.051 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj983 testdata.508946 2026-02-13T02:30:10.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:10.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj984 testdata.508946 2026-02-13T02:30:10.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:10.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj985 testdata.508946 2026-02-13T02:30:10.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:10.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj986 testdata.508946 2026-02-13T02:30:10.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:10.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj987 testdata.508946 2026-02-13T02:30:10.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:10.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj988 testdata.508946 2026-02-13T02:30:10.176 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:10.176 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj989 testdata.508946 2026-02-13T02:30:10.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:10.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj990 testdata.508946 2026-02-13T02:30:10.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:10.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj991 testdata.508946 2026-02-13T02:30:10.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:10.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj992 testdata.508946 2026-02-13T02:30:10.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:10.258 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj993 testdata.508946 2026-02-13T02:30:10.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:10.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj994 testdata.508946 2026-02-13T02:30:10.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:10.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj995 testdata.508946 2026-02-13T02:30:10.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:10.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj996 testdata.508946 2026-02-13T02:30:10.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:10.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj997 testdata.508946 2026-02-13T02:30:10.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:10.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj998 testdata.508946 2026-02-13T02:30:10.401 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:10.401 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj999 testdata.508946 2026-02-13T02:30:10.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:30:10.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj1000 testdata.508946 2026-02-13T02:30:10.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:334: _scrub_abort: rm -f testdata.508946 2026-02-13T02:30:10.452 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:336: _scrub_abort: get_primary test obj1 2026-02-13T02:30:10.452 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2026-02-13T02:30:10.452 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-13T02:30:10.452 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2026-02-13T02:30:10.453 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T02:30:10.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:336: _scrub_abort: local primary=1 2026-02-13T02:30:10.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: local pgid=1.0 2026-02-13T02:30:10.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: ceph tell 1.0 schedule-deep-scrub 2026-02-13T02:30:10.673 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:30:10.673 INFO:tasks.workunit.client.0.trial112.stdout: "deep": true, 2026-02-13T02:30:10.673 INFO:tasks.workunit.client.0.trial112.stdout: "must": false, 2026-02-13T02:30:10.674 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "2026-02-01T21:42:00.668644+0000" 2026-02-13T02:30:10.674 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:30:10.681 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:342: _scrub_abort: set -o pipefail 2026-02-13T02:30:10.681 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:343: _scrub_abort: found=no 2026-02-13T02:30:10.681 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:344: _scrub_abort: seq 0 200 2026-02-13T02:30:10.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:344: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:30:10.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:346: _scrub_abort: flush_pg_stats 2026-02-13T02:30:10.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:30:10.683 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:30:10.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:30:10.842 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:30:10.842 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:30:10.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:30:10.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:10.843 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:30:10.914 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836516 2026-02-13T02:30:10.915 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836516 2026-02-13T02:30:10.915 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836516' 2026-02-13T02:30:10.915 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:10.915 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:30:10.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705697 2026-02-13T02:30:10.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705697 2026-02-13T02:30:10.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836516 1-38654705697' 2026-02-13T02:30:10.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:10.989 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:30:11.058 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542175 2026-02-13T02:30:11.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542175 2026-02-13T02:30:11.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836516 1-38654705697 2-60129542175' 2026-02-13T02:30:11.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:11.059 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836516 2026-02-13T02:30:11.060 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:11.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:30:11.061 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836516 2026-02-13T02:30:11.061 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:11.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836516 2026-02-13T02:30:11.063 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836516 2026-02-13T02:30:11.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2026-02-13T02:30:11.063 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:11.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836514 -lt 21474836516 2026-02-13T02:30:11.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:30:12.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:30:12.236 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:12.397 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836517 -lt 21474836516 2026-02-13T02:30:12.397 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:12.398 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705697 2026-02-13T02:30:12.398 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:12.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:30:12.400 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705697 2026-02-13T02:30:12.400 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:12.401 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705697 2026-02-13T02:30:12.401 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705697 2026-02-13T02:30:12.402 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705697' 2026-02-13T02:30:12.402 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:30:12.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705698 -lt 38654705697 2026-02-13T02:30:12.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:12.563 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542175 2026-02-13T02:30:12.564 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:12.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:30:12.565 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542175 2026-02-13T02:30:12.566 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:12.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542175 2026-02-13T02:30:12.567 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542175 2026-02-13T02:30:12.568 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542175' 2026-02-13T02:30:12.568 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:30:12.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542176 -lt 60129542175 2026-02-13T02:30:12.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:347: _scrub_abort: ceph pg dump pgs 2026-02-13T02:30:12.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:347: _scrub_abort: grep '^1.0' 2026-02-13T02:30:12.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:347: _scrub_abort: grep -q scrubbing 2026-02-13T02:30:12.881 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:30:12.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:349: _scrub_abort: found=yes 2026-02-13T02:30:12.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:351: _scrub_abort: break 2026-02-13T02:30:12.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:354: _scrub_abort: set +o pipefail 2026-02-13T02:30:12.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:356: _scrub_abort: test yes = no 2026-02-13T02:30:12.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:362: _scrub_abort: ceph osd set nodeep-scrub 2026-02-13T02:30:13.097 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:30:13.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:363: _scrub_abort: '[' deep-scrub = deep-scrub ']' 2026-02-13T02:30:13.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:365: _scrub_abort: ceph osd set noscrub 2026-02-13T02:30:13.301 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:30:13.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:369: _scrub_abort: set -o pipefail 2026-02-13T02:30:13.313 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: seq 0 200 2026-02-13T02:30:13.315 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:30:13.315 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:30:13.315 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:30:13.316 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:30:13.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:30:13.477 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:30:13.477 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:30:13.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:30:13.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:13.477 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:30:13.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836520 2026-02-13T02:30:13.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836520 2026-02-13T02:30:13.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836520' 2026-02-13T02:30:13.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:13.550 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:30:13.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705701 2026-02-13T02:30:13.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705701 2026-02-13T02:30:13.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836520 1-38654705701' 2026-02-13T02:30:13.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:13.623 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:30:13.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542179 2026-02-13T02:30:13.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542179 2026-02-13T02:30:13.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836520 1-38654705701 2-60129542179' 2026-02-13T02:30:13.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:13.694 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836520 2026-02-13T02:30:13.694 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:13.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:30:13.696 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836520 2026-02-13T02:30:13.696 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:13.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836520 2026-02-13T02:30:13.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836520' 2026-02-13T02:30:13.698 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836520 2026-02-13T02:30:13.698 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:13.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836520 -lt 21474836520 2026-02-13T02:30:13.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:13.861 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705701 2026-02-13T02:30:13.861 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:13.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:30:13.863 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705701 2026-02-13T02:30:13.863 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:13.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705701 2026-02-13T02:30:13.865 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705701 2026-02-13T02:30:13.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705701' 2026-02-13T02:30:13.865 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:30:14.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705701 -lt 38654705701 2026-02-13T02:30:14.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:14.034 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542179 2026-02-13T02:30:14.034 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:14.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:30:14.036 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542179 2026-02-13T02:30:14.036 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:14.037 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542179 2026-02-13T02:30:14.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542179' 2026-02-13T02:30:14.038 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542179 2026-02-13T02:30:14.038 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:30:14.203 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542179 -lt 60129542179 2026-02-13T02:30:14.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:30:14.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:30:14.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:30:14.356 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:30:14.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:30:14.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:30:14.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:30:14.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:30:14.367 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:30:14.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:30:14.543 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:30:14.544 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:30:14.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:30:14.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:14.544 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:30:14.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836522 2026-02-13T02:30:14.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836522 2026-02-13T02:30:14.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836522' 2026-02-13T02:30:14.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:14.617 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:30:14.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705703 2026-02-13T02:30:14.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705703 2026-02-13T02:30:14.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836522 1-38654705703' 2026-02-13T02:30:14.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:14.690 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:30:14.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542181 2026-02-13T02:30:14.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542181 2026-02-13T02:30:14.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836522 1-38654705703 2-60129542181' 2026-02-13T02:30:14.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:14.762 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836522 2026-02-13T02:30:14.762 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:14.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:30:14.764 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836522 2026-02-13T02:30:14.764 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:14.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836522 2026-02-13T02:30:14.765 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836522 2026-02-13T02:30:14.766 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836522' 2026-02-13T02:30:14.766 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:14.925 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836520 -lt 21474836522 2026-02-13T02:30:14.925 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:30:15.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:30:15.937 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:16.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836523 -lt 21474836522 2026-02-13T02:30:16.089 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:16.089 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705703 2026-02-13T02:30:16.090 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:16.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:30:16.091 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705703 2026-02-13T02:30:16.092 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:16.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705703 2026-02-13T02:30:16.093 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705703 2026-02-13T02:30:16.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705703' 2026-02-13T02:30:16.094 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:30:16.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705704 -lt 38654705703 2026-02-13T02:30:16.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:16.264 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542181 2026-02-13T02:30:16.265 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:16.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:30:16.266 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542181 2026-02-13T02:30:16.266 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:16.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542181 2026-02-13T02:30:16.268 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542181 2026-02-13T02:30:16.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542181' 2026-02-13T02:30:16.269 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:30:16.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542182 -lt 60129542181 2026-02-13T02:30:16.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:30:16.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:30:16.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:30:16.573 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:30:16.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:30:16.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:30:16.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:30:16.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:30:16.583 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:30:16.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:30:16.738 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:30:16.738 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:30:16.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:30:16.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:16.738 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:30:16.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836525 2026-02-13T02:30:16.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836525 2026-02-13T02:30:16.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836525' 2026-02-13T02:30:16.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:16.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:30:16.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705706 2026-02-13T02:30:16.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705706 2026-02-13T02:30:16.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836525 1-38654705706' 2026-02-13T02:30:16.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:16.884 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:30:16.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542184 2026-02-13T02:30:16.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542184 2026-02-13T02:30:16.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836525 1-38654705706 2-60129542184' 2026-02-13T02:30:16.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:16.958 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836525 2026-02-13T02:30:16.958 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:16.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:30:16.959 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836525 2026-02-13T02:30:16.959 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:16.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836525 2026-02-13T02:30:16.961 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836525 2026-02-13T02:30:16.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836525' 2026-02-13T02:30:16.962 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:17.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836523 -lt 21474836525 2026-02-13T02:30:17.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:30:18.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:30:18.128 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:18.291 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836526 -lt 21474836525 2026-02-13T02:30:18.291 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:18.292 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705706 2026-02-13T02:30:18.292 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:18.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:30:18.294 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705706 2026-02-13T02:30:18.294 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:18.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705706 2026-02-13T02:30:18.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705706' 2026-02-13T02:30:18.296 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705706 2026-02-13T02:30:18.296 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:30:18.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705707 -lt 38654705706 2026-02-13T02:30:18.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:18.460 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542184 2026-02-13T02:30:18.460 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:18.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:30:18.461 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542184 2026-02-13T02:30:18.461 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:18.463 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542184 2026-02-13T02:30:18.463 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542184 2026-02-13T02:30:18.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542184' 2026-02-13T02:30:18.464 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:30:18.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542185 -lt 60129542184 2026-02-13T02:30:18.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:30:18.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:30:18.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:30:18.780 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:30:18.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:30:18.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:30:18.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:30:18.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:30:18.790 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:30:18.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:30:18.947 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:30:18.947 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:30:18.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:30:18.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:18.948 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:30:19.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836529 2026-02-13T02:30:19.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836529 2026-02-13T02:30:19.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836529' 2026-02-13T02:30:19.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:19.022 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:30:19.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705709 2026-02-13T02:30:19.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705709 2026-02-13T02:30:19.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836529 1-38654705709' 2026-02-13T02:30:19.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:19.094 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:30:19.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542187 2026-02-13T02:30:19.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542187 2026-02-13T02:30:19.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836529 1-38654705709 2-60129542187' 2026-02-13T02:30:19.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:19.166 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836529 2026-02-13T02:30:19.166 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:19.167 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:30:19.168 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836529 2026-02-13T02:30:19.168 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:19.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836529 2026-02-13T02:30:19.170 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836529 2026-02-13T02:30:19.170 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836529' 2026-02-13T02:30:19.170 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:19.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836526 -lt 21474836529 2026-02-13T02:30:19.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:30:20.339 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:30:20.339 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:20.502 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836529 -lt 21474836529 2026-02-13T02:30:20.502 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:20.502 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705709 2026-02-13T02:30:20.503 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:20.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:30:20.504 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705709 2026-02-13T02:30:20.505 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:20.506 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705709 2026-02-13T02:30:20.506 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705709 2026-02-13T02:30:20.507 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705709' 2026-02-13T02:30:20.507 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:30:20.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705710 -lt 38654705709 2026-02-13T02:30:20.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:20.678 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542187 2026-02-13T02:30:20.678 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:20.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:30:20.680 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542187 2026-02-13T02:30:20.680 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:20.681 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542187 2026-02-13T02:30:20.681 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542187 2026-02-13T02:30:20.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542187' 2026-02-13T02:30:20.682 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:30:20.848 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542188 -lt 60129542187 2026-02-13T02:30:20.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:30:20.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:30:20.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:30:21.001 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:30:21.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:30:21.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:30:21.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:30:21.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:30:21.012 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:30:21.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:30:21.169 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:30:21.169 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:30:21.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:30:21.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:21.170 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:30:21.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836532 2026-02-13T02:30:21.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836532 2026-02-13T02:30:21.242 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836532' 2026-02-13T02:30:21.242 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:21.242 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:30:21.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705713 2026-02-13T02:30:21.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705713 2026-02-13T02:30:21.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836532 1-38654705713' 2026-02-13T02:30:21.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:21.316 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:30:21.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542190 2026-02-13T02:30:21.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542190 2026-02-13T02:30:21.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836532 1-38654705713 2-60129542190' 2026-02-13T02:30:21.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:21.390 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836532 2026-02-13T02:30:21.390 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:21.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:30:21.392 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836532 2026-02-13T02:30:21.392 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:21.394 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836532 2026-02-13T02:30:21.394 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836532' 2026-02-13T02:30:21.394 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836532 2026-02-13T02:30:21.395 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:21.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836529 -lt 21474836532 2026-02-13T02:30:21.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:30:22.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:30:22.567 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:22.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836532 -lt 21474836532 2026-02-13T02:30:22.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:22.726 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705713 2026-02-13T02:30:22.726 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:22.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:30:22.728 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705713 2026-02-13T02:30:22.729 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:22.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705713 2026-02-13T02:30:22.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705713' 2026-02-13T02:30:22.730 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705713 2026-02-13T02:30:22.731 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:30:22.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705713 -lt 38654705713 2026-02-13T02:30:22.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:22.890 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542190 2026-02-13T02:30:22.890 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:22.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:30:22.892 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542190 2026-02-13T02:30:22.892 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:22.893 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542190 2026-02-13T02:30:22.894 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542190 2026-02-13T02:30:22.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542190' 2026-02-13T02:30:22.894 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:30:23.056 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542191 -lt 60129542190 2026-02-13T02:30:23.057 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:30:23.057 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:30:23.057 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:30:23.204 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:30:23.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:30:23.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:30:23.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:30:23.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:30:23.214 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:30:23.372 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:30:23.372 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:30:23.372 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:30:23.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:30:23.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:23.373 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:30:23.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836535 2026-02-13T02:30:23.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836535 2026-02-13T02:30:23.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836535' 2026-02-13T02:30:23.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:23.447 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:30:23.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705716 2026-02-13T02:30:23.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705716 2026-02-13T02:30:23.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836535 1-38654705716' 2026-02-13T02:30:23.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:23.519 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:30:23.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542193 2026-02-13T02:30:23.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542193 2026-02-13T02:30:23.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836535 1-38654705716 2-60129542193' 2026-02-13T02:30:23.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:23.596 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836535 2026-02-13T02:30:23.596 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:23.597 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:30:23.598 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836535 2026-02-13T02:30:23.598 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:23.599 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836535 2026-02-13T02:30:23.599 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836535' 2026-02-13T02:30:23.600 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836535 2026-02-13T02:30:23.600 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:23.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836535 -lt 21474836535 2026-02-13T02:30:23.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:23.763 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705716 2026-02-13T02:30:23.763 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:23.764 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:30:23.765 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705716 2026-02-13T02:30:23.765 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:23.767 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705716 2026-02-13T02:30:23.767 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705716 2026-02-13T02:30:23.768 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705716' 2026-02-13T02:30:23.768 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:30:23.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705716 -lt 38654705716 2026-02-13T02:30:23.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:23.936 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542193 2026-02-13T02:30:23.936 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:23.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:30:23.938 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542193 2026-02-13T02:30:23.938 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:23.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542193 2026-02-13T02:30:23.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542193' 2026-02-13T02:30:23.940 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542193 2026-02-13T02:30:23.940 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:30:24.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542194 -lt 60129542193 2026-02-13T02:30:24.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:30:24.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:30:24.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:30:24.247 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:30:24.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:30:24.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:30:24.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:30:24.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:30:24.257 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:30:24.418 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:30:24.419 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:30:24.419 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:30:24.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:30:24.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:24.419 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:30:24.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836537 2026-02-13T02:30:24.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836537 2026-02-13T02:30:24.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836537' 2026-02-13T02:30:24.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:24.492 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:30:24.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705718 2026-02-13T02:30:24.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705718 2026-02-13T02:30:24.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836537 1-38654705718' 2026-02-13T02:30:24.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:24.564 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:30:24.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542196 2026-02-13T02:30:24.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542196 2026-02-13T02:30:24.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836537 1-38654705718 2-60129542196' 2026-02-13T02:30:24.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:24.634 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836537 2026-02-13T02:30:24.635 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:24.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:30:24.636 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836537 2026-02-13T02:30:24.636 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:24.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836537 2026-02-13T02:30:24.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836537' 2026-02-13T02:30:24.638 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836537 2026-02-13T02:30:24.639 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:24.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836535 -lt 21474836537 2026-02-13T02:30:24.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:30:25.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:30:25.804 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:25.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836538 -lt 21474836537 2026-02-13T02:30:25.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:25.969 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705718 2026-02-13T02:30:25.969 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:25.970 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:30:25.971 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705718 2026-02-13T02:30:25.971 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:25.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705718 2026-02-13T02:30:25.973 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705718 2026-02-13T02:30:25.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705718' 2026-02-13T02:30:25.973 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:30:26.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705719 -lt 38654705718 2026-02-13T02:30:26.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:26.140 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542196 2026-02-13T02:30:26.140 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:26.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:30:26.143 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542196 2026-02-13T02:30:26.143 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:26.144 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542196 2026-02-13T02:30:26.144 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542196 2026-02-13T02:30:26.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542196' 2026-02-13T02:30:26.145 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:30:26.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542197 -lt 60129542196 2026-02-13T02:30:26.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:30:26.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:30:26.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:30:26.453 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:30:26.463 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:30:26.463 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:30:26.463 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:30:26.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:30:26.464 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:30:26.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:30:26.623 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:30:26.623 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:30:26.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:30:26.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:26.624 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:30:26.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836540 2026-02-13T02:30:26.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836540 2026-02-13T02:30:26.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836540' 2026-02-13T02:30:26.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:26.697 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:30:26.769 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705721 2026-02-13T02:30:26.770 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705721 2026-02-13T02:30:26.770 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836540 1-38654705721' 2026-02-13T02:30:26.770 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:26.770 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:30:26.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542199 2026-02-13T02:30:26.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542199 2026-02-13T02:30:26.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836540 1-38654705721 2-60129542199' 2026-02-13T02:30:26.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:26.842 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836540 2026-02-13T02:30:26.842 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:26.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:30:26.843 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836540 2026-02-13T02:30:26.844 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:26.845 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836540 2026-02-13T02:30:26.845 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836540 2026-02-13T02:30:26.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836540' 2026-02-13T02:30:26.846 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:27.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836538 -lt 21474836540 2026-02-13T02:30:27.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:30:28.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:30:28.011 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:28.175 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836541 -lt 21474836540 2026-02-13T02:30:28.175 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:28.176 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705721 2026-02-13T02:30:28.176 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:28.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:30:28.178 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705721 2026-02-13T02:30:28.178 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:28.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705721 2026-02-13T02:30:28.180 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705721 2026-02-13T02:30:28.180 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705721' 2026-02-13T02:30:28.180 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:30:28.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705722 -lt 38654705721 2026-02-13T02:30:28.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:28.339 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542199 2026-02-13T02:30:28.339 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:28.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:30:28.340 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542199 2026-02-13T02:30:28.341 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:28.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542199 2026-02-13T02:30:28.342 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542199 2026-02-13T02:30:28.343 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542199' 2026-02-13T02:30:28.343 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:30:28.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542200 -lt 60129542199 2026-02-13T02:30:28.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:30:28.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:30:28.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:30:28.670 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:30:28.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:30:28.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:30:28.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:30:28.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:30:28.683 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:30:28.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:30:28.844 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:30:28.844 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:30:28.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:30:28.845 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:28.845 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:30:28.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836543 2026-02-13T02:30:28.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836543 2026-02-13T02:30:28.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836543' 2026-02-13T02:30:28.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:28.917 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:30:28.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705724 2026-02-13T02:30:28.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705724 2026-02-13T02:30:28.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836543 1-38654705724' 2026-02-13T02:30:28.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:28.989 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:30:29.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542202 2026-02-13T02:30:29.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542202 2026-02-13T02:30:29.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836543 1-38654705724 2-60129542202' 2026-02-13T02:30:29.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:29.064 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836543 2026-02-13T02:30:29.064 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:29.065 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:30:29.066 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836543 2026-02-13T02:30:29.066 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:29.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836543 2026-02-13T02:30:29.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836543' 2026-02-13T02:30:29.068 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836543 2026-02-13T02:30:29.068 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:29.230 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836541 -lt 21474836543 2026-02-13T02:30:29.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:30:30.232 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:30:30.232 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:30.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836544 -lt 21474836543 2026-02-13T02:30:30.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:30.393 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705724 2026-02-13T02:30:30.393 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:30.394 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:30:30.395 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705724 2026-02-13T02:30:30.395 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:30.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705724 2026-02-13T02:30:30.397 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705724 2026-02-13T02:30:30.397 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705724' 2026-02-13T02:30:30.397 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:30:30.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705725 -lt 38654705724 2026-02-13T02:30:30.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:30.561 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542202 2026-02-13T02:30:30.561 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:30.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:30:30.563 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542202 2026-02-13T02:30:30.563 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:30.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542202 2026-02-13T02:30:30.564 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542202 2026-02-13T02:30:30.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542202' 2026-02-13T02:30:30.565 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:30:30.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542203 -lt 60129542202 2026-02-13T02:30:30.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:30:30.718 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:30:30.718 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:30:30.871 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:30:30.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:30:30.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:30:30.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:30:30.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:30:30.881 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:30:31.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:30:31.039 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:30:31.040 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:30:31.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:30:31.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:31.040 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:30:31.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836547 2026-02-13T02:30:31.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836547 2026-02-13T02:30:31.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836547' 2026-02-13T02:30:31.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:31.117 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:30:31.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705728 2026-02-13T02:30:31.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705728 2026-02-13T02:30:31.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836547 1-38654705728' 2026-02-13T02:30:31.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:31.188 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:30:31.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542205 2026-02-13T02:30:31.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542205 2026-02-13T02:30:31.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836547 1-38654705728 2-60129542205' 2026-02-13T02:30:31.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:31.264 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836547 2026-02-13T02:30:31.264 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:31.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:30:31.265 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836547 2026-02-13T02:30:31.266 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:31.267 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836547 2026-02-13T02:30:31.267 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836547' 2026-02-13T02:30:31.267 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836547 2026-02-13T02:30:31.268 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:31.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836544 -lt 21474836547 2026-02-13T02:30:31.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:30:32.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:30:32.424 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:32.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836547 -lt 21474836547 2026-02-13T02:30:32.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:32.583 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705728 2026-02-13T02:30:32.583 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:32.584 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:30:32.585 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705728 2026-02-13T02:30:32.585 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:32.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705728 2026-02-13T02:30:32.587 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705728 2026-02-13T02:30:32.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705728' 2026-02-13T02:30:32.587 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:30:32.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705728 -lt 38654705728 2026-02-13T02:30:32.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:32.760 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542205 2026-02-13T02:30:32.760 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:32.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:30:32.762 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542205 2026-02-13T02:30:32.762 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:32.764 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542205 2026-02-13T02:30:32.764 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542205' 2026-02-13T02:30:32.764 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542205 2026-02-13T02:30:32.764 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:30:32.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542206 -lt 60129542205 2026-02-13T02:30:32.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:30:32.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:30:32.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:30:33.068 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:30:33.079 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:30:33.079 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:30:33.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:30:33.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:30:33.080 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:30:33.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:30:33.239 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:30:33.239 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:30:33.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:30:33.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:33.239 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:30:33.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836550 2026-02-13T02:30:33.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836550 2026-02-13T02:30:33.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836550' 2026-02-13T02:30:33.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:33.311 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:30:33.382 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705731 2026-02-13T02:30:33.382 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705731 2026-02-13T02:30:33.382 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836550 1-38654705731' 2026-02-13T02:30:33.382 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:33.383 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:30:33.455 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542208 2026-02-13T02:30:33.456 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542208 2026-02-13T02:30:33.456 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836550 1-38654705731 2-60129542208' 2026-02-13T02:30:33.456 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:33.456 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836550 2026-02-13T02:30:33.457 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:33.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:30:33.458 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836550 2026-02-13T02:30:33.458 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:33.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836550 2026-02-13T02:30:33.460 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836550 2026-02-13T02:30:33.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836550' 2026-02-13T02:30:33.460 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:33.618 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836547 -lt 21474836550 2026-02-13T02:30:33.618 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:30:34.619 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:30:34.620 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:34.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836550 -lt 21474836550 2026-02-13T02:30:34.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:34.778 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705731 2026-02-13T02:30:34.778 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:34.779 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:30:34.780 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705731 2026-02-13T02:30:34.780 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:34.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705731 2026-02-13T02:30:34.781 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705731 2026-02-13T02:30:34.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705731' 2026-02-13T02:30:34.782 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:30:34.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705731 -lt 38654705731 2026-02-13T02:30:34.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:34.941 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542208 2026-02-13T02:30:34.942 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:34.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:30:34.943 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542208 2026-02-13T02:30:34.943 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:34.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542208 2026-02-13T02:30:34.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542208' 2026-02-13T02:30:34.945 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542208 2026-02-13T02:30:34.945 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:30:35.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542209 -lt 60129542208 2026-02-13T02:30:35.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:30:35.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:30:35.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:30:35.259 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:30:35.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:30:35.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:30:35.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:30:35.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:30:35.268 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:30:35.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:30:35.423 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:30:35.423 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:30:35.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:30:35.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:35.424 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:30:35.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836553 2026-02-13T02:30:35.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836553 2026-02-13T02:30:35.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836553' 2026-02-13T02:30:35.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:35.495 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:30:35.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705734 2026-02-13T02:30:35.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705734 2026-02-13T02:30:35.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836553 1-38654705734' 2026-02-13T02:30:35.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:35.566 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:30:35.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542212 2026-02-13T02:30:35.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542212 2026-02-13T02:30:35.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836553 1-38654705734 2-60129542212' 2026-02-13T02:30:35.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:35.634 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836553 2026-02-13T02:30:35.634 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:35.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:30:35.636 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836553 2026-02-13T02:30:35.636 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:35.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836553 2026-02-13T02:30:35.638 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836553 2026-02-13T02:30:35.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836553' 2026-02-13T02:30:35.638 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:35.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836553 -lt 21474836553 2026-02-13T02:30:35.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:35.805 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705734 2026-02-13T02:30:35.805 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:35.806 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:30:35.807 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705734 2026-02-13T02:30:35.807 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:35.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705734 2026-02-13T02:30:35.809 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705734 2026-02-13T02:30:35.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705734' 2026-02-13T02:30:35.809 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:30:35.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705734 -lt 38654705734 2026-02-13T02:30:35.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:35.973 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542212 2026-02-13T02:30:35.973 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:35.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:30:35.975 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542212 2026-02-13T02:30:35.975 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:35.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542212 2026-02-13T02:30:35.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542212' 2026-02-13T02:30:35.977 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542212 2026-02-13T02:30:35.977 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:30:36.144 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542212 -lt 60129542212 2026-02-13T02:30:36.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:30:36.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:30:36.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:30:36.293 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:30:36.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:30:36.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:30:36.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:30:36.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:30:36.305 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:30:36.465 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:30:36.465 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:30:36.465 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:30:36.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:30:36.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:36.466 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:30:36.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836555 2026-02-13T02:30:36.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836555 2026-02-13T02:30:36.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836555' 2026-02-13T02:30:36.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:36.538 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:30:36.611 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705736 2026-02-13T02:30:36.611 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705736 2026-02-13T02:30:36.611 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836555 1-38654705736' 2026-02-13T02:30:36.611 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:36.611 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:30:36.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542214 2026-02-13T02:30:36.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542214 2026-02-13T02:30:36.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836555 1-38654705736 2-60129542214' 2026-02-13T02:30:36.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:36.684 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836555 2026-02-13T02:30:36.685 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:36.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:30:36.686 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836555 2026-02-13T02:30:36.686 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:36.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836555 2026-02-13T02:30:36.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836555' 2026-02-13T02:30:36.688 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836555 2026-02-13T02:30:36.688 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:36.847 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836553 -lt 21474836555 2026-02-13T02:30:36.848 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:30:37.848 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:30:37.849 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:38.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836556 -lt 21474836555 2026-02-13T02:30:38.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:38.014 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705736 2026-02-13T02:30:38.014 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:38.015 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:30:38.016 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705736 2026-02-13T02:30:38.016 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:38.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705736 2026-02-13T02:30:38.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705736' 2026-02-13T02:30:38.018 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705736 2026-02-13T02:30:38.018 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:30:38.180 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705737 -lt 38654705736 2026-02-13T02:30:38.181 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:38.181 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542214 2026-02-13T02:30:38.181 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:38.182 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:30:38.183 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542214 2026-02-13T02:30:38.183 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:38.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542214 2026-02-13T02:30:38.185 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542214 2026-02-13T02:30:38.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542214' 2026-02-13T02:30:38.186 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:30:38.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542215 -lt 60129542214 2026-02-13T02:30:38.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:30:38.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:30:38.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:30:38.494 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:30:38.505 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:30:38.505 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:30:38.506 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:30:38.506 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:30:38.506 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:30:38.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:30:38.669 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:30:38.669 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:30:38.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:30:38.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:38.669 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:30:38.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836558 2026-02-13T02:30:38.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836558 2026-02-13T02:30:38.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836558' 2026-02-13T02:30:38.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:38.741 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:30:38.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705739 2026-02-13T02:30:38.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705739 2026-02-13T02:30:38.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836558 1-38654705739' 2026-02-13T02:30:38.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:38.814 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:30:38.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542217 2026-02-13T02:30:38.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542217 2026-02-13T02:30:38.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836558 1-38654705739 2-60129542217' 2026-02-13T02:30:38.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:38.889 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836558 2026-02-13T02:30:38.889 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:38.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:30:38.890 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836558 2026-02-13T02:30:38.891 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:38.892 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836558 2026-02-13T02:30:38.892 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836558' 2026-02-13T02:30:38.892 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836558 2026-02-13T02:30:38.893 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:39.053 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836556 -lt 21474836558 2026-02-13T02:30:39.053 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:30:40.054 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:30:40.055 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:40.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836559 -lt 21474836558 2026-02-13T02:30:40.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:40.214 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705739 2026-02-13T02:30:40.214 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:40.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:30:40.216 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705739 2026-02-13T02:30:40.216 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:40.217 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705739 2026-02-13T02:30:40.218 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705739 2026-02-13T02:30:40.218 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705739' 2026-02-13T02:30:40.218 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:30:40.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705740 -lt 38654705739 2026-02-13T02:30:40.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:40.388 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542217 2026-02-13T02:30:40.389 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:40.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:30:40.390 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542217 2026-02-13T02:30:40.390 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:40.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542217 2026-02-13T02:30:40.392 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542217 2026-02-13T02:30:40.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542217' 2026-02-13T02:30:40.393 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:30:40.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542218 -lt 60129542217 2026-02-13T02:30:40.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:30:40.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:30:40.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:30:40.703 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:30:40.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:30:40.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:30:40.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:30:40.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:30:40.714 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:30:40.871 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:30:40.871 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:30:40.871 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:30:40.871 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:30:40.872 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:40.872 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:30:40.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836561 2026-02-13T02:30:40.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836561 2026-02-13T02:30:40.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836561' 2026-02-13T02:30:40.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:40.945 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:30:41.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705742 2026-02-13T02:30:41.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705742 2026-02-13T02:30:41.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836561 1-38654705742' 2026-02-13T02:30:41.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:41.019 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:30:41.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542220 2026-02-13T02:30:41.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542220 2026-02-13T02:30:41.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836561 1-38654705742 2-60129542220' 2026-02-13T02:30:41.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:41.093 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836561 2026-02-13T02:30:41.093 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:41.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:30:41.094 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836561 2026-02-13T02:30:41.095 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:41.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836561 2026-02-13T02:30:41.096 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836561 2026-02-13T02:30:41.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836561' 2026-02-13T02:30:41.097 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:41.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836559 -lt 21474836561 2026-02-13T02:30:41.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:30:42.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:30:42.263 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:42.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836562 -lt 21474836561 2026-02-13T02:30:42.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:42.434 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705742 2026-02-13T02:30:42.434 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:42.435 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:30:42.436 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705742 2026-02-13T02:30:42.436 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:42.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705742 2026-02-13T02:30:42.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705742' 2026-02-13T02:30:42.438 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705742 2026-02-13T02:30:42.438 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:30:42.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705743 -lt 38654705742 2026-02-13T02:30:42.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:42.606 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542220 2026-02-13T02:30:42.606 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:42.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:30:42.608 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542220 2026-02-13T02:30:42.608 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:42.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542220 2026-02-13T02:30:42.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542220' 2026-02-13T02:30:42.610 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542220 2026-02-13T02:30:42.610 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:30:42.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542221 -lt 60129542220 2026-02-13T02:30:42.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:30:42.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:30:42.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:30:42.930 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:30:42.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:30:42.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:30:42.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:30:42.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:30:42.942 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:30:43.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:30:43.100 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:30:43.100 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:30:43.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:30:43.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:43.101 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:30:43.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836565 2026-02-13T02:30:43.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836565 2026-02-13T02:30:43.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836565' 2026-02-13T02:30:43.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:43.174 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:30:43.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705746 2026-02-13T02:30:43.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705746 2026-02-13T02:30:43.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836565 1-38654705746' 2026-02-13T02:30:43.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:43.250 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:30:43.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542223 2026-02-13T02:30:43.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542223 2026-02-13T02:30:43.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836565 1-38654705746 2-60129542223' 2026-02-13T02:30:43.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:43.322 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836565 2026-02-13T02:30:43.322 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:43.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:30:43.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836565 2026-02-13T02:30:43.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:43.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836565 2026-02-13T02:30:43.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836565' 2026-02-13T02:30:43.325 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836565 2026-02-13T02:30:43.325 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:43.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836562 -lt 21474836565 2026-02-13T02:30:43.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:30:44.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:30:44.500 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:44.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836565 -lt 21474836565 2026-02-13T02:30:44.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:44.658 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705746 2026-02-13T02:30:44.658 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:44.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:30:44.660 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705746 2026-02-13T02:30:44.660 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:44.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705746 2026-02-13T02:30:44.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705746' 2026-02-13T02:30:44.662 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705746 2026-02-13T02:30:44.662 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:30:44.835 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705746 -lt 38654705746 2026-02-13T02:30:44.835 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:44.836 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542223 2026-02-13T02:30:44.836 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:44.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:30:44.838 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542223 2026-02-13T02:30:44.838 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:44.839 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542223 2026-02-13T02:30:44.839 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542223' 2026-02-13T02:30:44.840 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542223 2026-02-13T02:30:44.840 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:30:44.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542224 -lt 60129542223 2026-02-13T02:30:45.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:30:45.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:30:45.001 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:30:45.152 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:30:45.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:30:45.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:30:45.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:30:45.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:30:45.164 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:30:45.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:30:45.323 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:30:45.323 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:30:45.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:30:45.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:45.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:30:45.397 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836568 2026-02-13T02:30:45.397 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836568 2026-02-13T02:30:45.397 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836568' 2026-02-13T02:30:45.397 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:45.397 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:30:45.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705749 2026-02-13T02:30:45.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705749 2026-02-13T02:30:45.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836568 1-38654705749' 2026-02-13T02:30:45.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:45.471 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:30:45.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542226 2026-02-13T02:30:45.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542226 2026-02-13T02:30:45.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836568 1-38654705749 2-60129542226' 2026-02-13T02:30:45.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:45.543 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836568 2026-02-13T02:30:45.543 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:45.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:30:45.544 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836568 2026-02-13T02:30:45.544 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:45.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836568 2026-02-13T02:30:45.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836568' 2026-02-13T02:30:45.546 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836568 2026-02-13T02:30:45.547 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:45.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836565 -lt 21474836568 2026-02-13T02:30:45.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:30:46.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:30:46.710 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:46.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836568 -lt 21474836568 2026-02-13T02:30:46.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:46.875 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705749 2026-02-13T02:30:46.875 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:46.876 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:30:46.877 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705749 2026-02-13T02:30:46.877 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:46.878 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705749 2026-02-13T02:30:46.878 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705749 2026-02-13T02:30:46.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705749' 2026-02-13T02:30:46.879 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:30:47.051 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705749 -lt 38654705749 2026-02-13T02:30:47.051 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:47.051 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542226 2026-02-13T02:30:47.052 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:47.052 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:30:47.053 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542226 2026-02-13T02:30:47.053 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:47.055 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542226 2026-02-13T02:30:47.055 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542226 2026-02-13T02:30:47.055 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542226' 2026-02-13T02:30:47.056 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:30:47.222 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542227 -lt 60129542226 2026-02-13T02:30:47.223 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:30:47.223 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:30:47.223 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:30:47.368 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:30:47.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:30:47.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:30:47.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:30:47.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:30:47.378 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:30:47.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:30:47.533 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:30:47.533 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:30:47.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:30:47.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:47.533 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:30:47.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836571 2026-02-13T02:30:47.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836571 2026-02-13T02:30:47.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836571' 2026-02-13T02:30:47.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:47.607 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:30:47.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705752 2026-02-13T02:30:47.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705752 2026-02-13T02:30:47.680 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836571 1-38654705752' 2026-02-13T02:30:47.680 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:47.680 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:30:47.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542230 2026-02-13T02:30:47.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542230 2026-02-13T02:30:47.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836571 1-38654705752 2-60129542230' 2026-02-13T02:30:47.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:47.753 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836571 2026-02-13T02:30:47.753 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:47.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:30:47.755 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836571 2026-02-13T02:30:47.755 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:47.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836571 2026-02-13T02:30:47.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836571' 2026-02-13T02:30:47.756 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836571 2026-02-13T02:30:47.757 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:47.915 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836571 -lt 21474836571 2026-02-13T02:30:47.915 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:47.916 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705752 2026-02-13T02:30:47.916 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:47.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:30:47.918 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705752 2026-02-13T02:30:47.918 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:47.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705752 2026-02-13T02:30:47.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705752' 2026-02-13T02:30:47.920 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705752 2026-02-13T02:30:47.920 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:30:48.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705752 -lt 38654705752 2026-02-13T02:30:48.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:48.085 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542230 2026-02-13T02:30:48.085 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:48.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:30:48.086 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542230 2026-02-13T02:30:48.086 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:48.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542230 2026-02-13T02:30:48.088 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542230 2026-02-13T02:30:48.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542230' 2026-02-13T02:30:48.088 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:30:48.243 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542229 -lt 60129542230 2026-02-13T02:30:48.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:30:49.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:30:49.245 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:30:49.410 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542229 -lt 60129542230 2026-02-13T02:30:49.410 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:30:50.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:30:50.412 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:30:50.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542232 -lt 60129542230 2026-02-13T02:30:50.572 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:30:50.572 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:30:50.572 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:30:50.720 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:30:50.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:30:50.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:30:50.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:30:50.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:30:50.731 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:30:50.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:30:50.889 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:30:50.889 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:30:50.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:30:50.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:50.890 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:30:50.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836575 2026-02-13T02:30:50.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836575 2026-02-13T02:30:50.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836575' 2026-02-13T02:30:50.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:50.964 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:30:51.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705756 2026-02-13T02:30:51.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705756 2026-02-13T02:30:51.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836575 1-38654705756' 2026-02-13T02:30:51.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:51.041 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:30:51.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542234 2026-02-13T02:30:51.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542234 2026-02-13T02:30:51.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836575 1-38654705756 2-60129542234' 2026-02-13T02:30:51.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:51.114 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836575 2026-02-13T02:30:51.115 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:51.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:30:51.116 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836575 2026-02-13T02:30:51.116 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:51.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836575 2026-02-13T02:30:51.118 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836575 2026-02-13T02:30:51.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836575' 2026-02-13T02:30:51.119 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:51.280 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836573 -lt 21474836575 2026-02-13T02:30:51.280 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:30:52.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:30:52.282 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:52.442 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836576 -lt 21474836575 2026-02-13T02:30:52.442 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:52.442 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705756 2026-02-13T02:30:52.443 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:52.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:30:52.444 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705756 2026-02-13T02:30:52.444 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:52.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705756 2026-02-13T02:30:52.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705756' 2026-02-13T02:30:52.447 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705756 2026-02-13T02:30:52.447 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:30:52.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705757 -lt 38654705756 2026-02-13T02:30:52.615 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:52.615 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542234 2026-02-13T02:30:52.616 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:52.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:30:52.617 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542234 2026-02-13T02:30:52.617 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:52.618 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542234 2026-02-13T02:30:52.619 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542234' 2026-02-13T02:30:52.619 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542234 2026-02-13T02:30:52.619 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:30:52.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542235 -lt 60129542234 2026-02-13T02:30:52.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:30:52.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:30:52.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:30:52.929 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:30:52.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:30:52.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:30:52.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:30:52.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:30:52.939 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:30:53.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:30:53.101 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:30:53.101 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:30:53.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:30:53.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:53.101 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:30:53.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836579 2026-02-13T02:30:53.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836579 2026-02-13T02:30:53.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836579' 2026-02-13T02:30:53.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:53.174 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:30:53.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705760 2026-02-13T02:30:53.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705760 2026-02-13T02:30:53.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836579 1-38654705760' 2026-02-13T02:30:53.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:53.246 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:30:53.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542237 2026-02-13T02:30:53.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542237 2026-02-13T02:30:53.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836579 1-38654705760 2-60129542237' 2026-02-13T02:30:53.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:53.317 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836579 2026-02-13T02:30:53.318 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:53.319 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:30:53.320 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836579 2026-02-13T02:30:53.320 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:53.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836579 2026-02-13T02:30:53.321 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836579 2026-02-13T02:30:53.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836579' 2026-02-13T02:30:53.321 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:53.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836576 -lt 21474836579 2026-02-13T02:30:53.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:30:54.494 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:30:54.495 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:54.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836579 -lt 21474836579 2026-02-13T02:30:54.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:54.654 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705760 2026-02-13T02:30:54.654 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:54.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:30:54.656 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705760 2026-02-13T02:30:54.656 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:54.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705760 2026-02-13T02:30:54.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705760' 2026-02-13T02:30:54.657 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705760 2026-02-13T02:30:54.658 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:30:54.827 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705760 -lt 38654705760 2026-02-13T02:30:54.827 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:54.828 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542237 2026-02-13T02:30:54.828 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:54.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:30:54.830 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542237 2026-02-13T02:30:54.830 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:54.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542237 2026-02-13T02:30:54.832 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542237 2026-02-13T02:30:54.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542237' 2026-02-13T02:30:54.832 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:30:54.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542238 -lt 60129542237 2026-02-13T02:30:54.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:30:54.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:30:54.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:30:55.146 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:30:55.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:30:55.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:30:55.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:30:55.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:30:55.156 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:30:55.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:30:55.311 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:30:55.311 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:30:55.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:30:55.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:55.311 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:30:55.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836582 2026-02-13T02:30:55.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836582 2026-02-13T02:30:55.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836582' 2026-02-13T02:30:55.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:55.387 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:30:55.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705763 2026-02-13T02:30:55.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705763 2026-02-13T02:30:55.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836582 1-38654705763' 2026-02-13T02:30:55.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:55.460 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:30:55.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542240 2026-02-13T02:30:55.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542240 2026-02-13T02:30:55.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836582 1-38654705763 2-60129542240' 2026-02-13T02:30:55.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:55.535 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836582 2026-02-13T02:30:55.535 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:55.536 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:30:55.536 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836582 2026-02-13T02:30:55.536 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:55.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836582 2026-02-13T02:30:55.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836582' 2026-02-13T02:30:55.538 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836582 2026-02-13T02:30:55.538 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:55.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836579 -lt 21474836582 2026-02-13T02:30:55.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:30:56.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:30:56.702 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:56.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836582 -lt 21474836582 2026-02-13T02:30:56.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:56.861 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705763 2026-02-13T02:30:56.861 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:56.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:30:56.863 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705763 2026-02-13T02:30:56.863 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:56.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705763 2026-02-13T02:30:56.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705763' 2026-02-13T02:30:56.864 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705763 2026-02-13T02:30:56.865 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:30:57.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705763 -lt 38654705763 2026-02-13T02:30:57.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:57.037 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542240 2026-02-13T02:30:57.037 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:57.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:30:57.039 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542240 2026-02-13T02:30:57.039 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:57.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542240 2026-02-13T02:30:57.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542240' 2026-02-13T02:30:57.041 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542240 2026-02-13T02:30:57.041 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:30:57.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542241 -lt 60129542240 2026-02-13T02:30:57.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:30:57.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:30:57.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:30:57.361 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:30:57.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:30:57.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:30:57.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:30:57.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:30:57.371 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:30:57.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:30:57.529 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:30:57.529 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:30:57.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:30:57.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:57.529 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:30:57.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836585 2026-02-13T02:30:57.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836585 2026-02-13T02:30:57.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836585' 2026-02-13T02:30:57.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:57.602 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:30:57.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705766 2026-02-13T02:30:57.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705766 2026-02-13T02:30:57.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836585 1-38654705766' 2026-02-13T02:30:57.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:30:57.674 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:30:57.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542244 2026-02-13T02:30:57.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542244 2026-02-13T02:30:57.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836585 1-38654705766 2-60129542244' 2026-02-13T02:30:57.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:57.746 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836585 2026-02-13T02:30:57.746 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:57.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:30:57.747 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836585 2026-02-13T02:30:57.747 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:57.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836585 2026-02-13T02:30:57.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836585' 2026-02-13T02:30:57.749 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836585 2026-02-13T02:30:57.750 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:30:57.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836585 -lt 21474836585 2026-02-13T02:30:57.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:57.917 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705766 2026-02-13T02:30:57.918 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:57.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:30:57.919 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705766 2026-02-13T02:30:57.920 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:57.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705766 2026-02-13T02:30:57.921 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705766 2026-02-13T02:30:57.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705766' 2026-02-13T02:30:57.922 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:30:58.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705766 -lt 38654705766 2026-02-13T02:30:58.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:30:58.088 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542244 2026-02-13T02:30:58.088 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:30:58.089 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:30:58.090 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542244 2026-02-13T02:30:58.090 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:30:58.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542244 2026-02-13T02:30:58.092 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542244 2026-02-13T02:30:58.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542244' 2026-02-13T02:30:58.092 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:30:58.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542243 -lt 60129542244 2026-02-13T02:30:58.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:30:59.253 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:30:59.254 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:30:59.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542243 -lt 60129542244 2026-02-13T02:30:59.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:31:00.416 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:31:00.416 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:31:00.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542246 -lt 60129542244 2026-02-13T02:31:00.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:31:00.572 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:31:00.572 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:31:00.723 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:31:00.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:31:00.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:31:00.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:31:00.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:31:00.733 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:31:00.892 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:31:00.892 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:31:00.892 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:31:00.892 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:31:00.893 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:00.893 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:31:00.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836589 2026-02-13T02:31:00.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836589 2026-02-13T02:31:00.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836589' 2026-02-13T02:31:00.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:00.966 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:31:01.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705770 2026-02-13T02:31:01.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705770 2026-02-13T02:31:01.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836589 1-38654705770' 2026-02-13T02:31:01.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:01.038 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:31:01.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542248 2026-02-13T02:31:01.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542248 2026-02-13T02:31:01.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836589 1-38654705770 2-60129542248' 2026-02-13T02:31:01.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:01.110 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836589 2026-02-13T02:31:01.111 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:01.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:31:01.112 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836589 2026-02-13T02:31:01.113 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:01.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836589 2026-02-13T02:31:01.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836589' 2026-02-13T02:31:01.114 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836589 2026-02-13T02:31:01.115 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:01.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836587 -lt 21474836589 2026-02-13T02:31:01.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:31:02.280 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:31:02.281 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:02.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836590 -lt 21474836589 2026-02-13T02:31:02.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:02.445 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705770 2026-02-13T02:31:02.445 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:02.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:31:02.447 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705770 2026-02-13T02:31:02.448 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:02.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705770 2026-02-13T02:31:02.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705770' 2026-02-13T02:31:02.449 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705770 2026-02-13T02:31:02.449 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:31:02.611 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705771 -lt 38654705770 2026-02-13T02:31:02.611 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:02.611 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542248 2026-02-13T02:31:02.612 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:02.613 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:31:02.613 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542248 2026-02-13T02:31:02.614 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:02.615 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542248 2026-02-13T02:31:02.615 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542248' 2026-02-13T02:31:02.616 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542248 2026-02-13T02:31:02.616 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:31:02.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542249 -lt 60129542248 2026-02-13T02:31:02.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:31:02.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:31:02.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:31:02.932 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:31:02.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:31:02.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:31:02.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:31:02.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:31:02.942 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:31:03.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:31:03.097 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:31:03.097 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:31:03.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:31:03.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:03.097 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:31:03.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836593 2026-02-13T02:31:03.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836593 2026-02-13T02:31:03.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836593' 2026-02-13T02:31:03.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:03.171 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:31:03.242 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705774 2026-02-13T02:31:03.242 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705774 2026-02-13T02:31:03.242 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836593 1-38654705774' 2026-02-13T02:31:03.242 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:03.242 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:31:03.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542251 2026-02-13T02:31:03.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542251 2026-02-13T02:31:03.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836593 1-38654705774 2-60129542251' 2026-02-13T02:31:03.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:03.317 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836593 2026-02-13T02:31:03.317 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:03.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:31:03.319 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836593 2026-02-13T02:31:03.319 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:03.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836593 2026-02-13T02:31:03.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836593' 2026-02-13T02:31:03.320 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836593 2026-02-13T02:31:03.321 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:03.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836590 -lt 21474836593 2026-02-13T02:31:03.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:31:04.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:31:04.485 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:04.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836593 -lt 21474836593 2026-02-13T02:31:04.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:04.653 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705774 2026-02-13T02:31:04.653 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:04.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:31:04.655 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705774 2026-02-13T02:31:04.655 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:04.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705774 2026-02-13T02:31:04.656 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705774 2026-02-13T02:31:04.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705774' 2026-02-13T02:31:04.657 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:31:04.823 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705774 -lt 38654705774 2026-02-13T02:31:04.823 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:04.824 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542251 2026-02-13T02:31:04.824 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:04.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:31:04.826 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542251 2026-02-13T02:31:04.826 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:04.827 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542251 2026-02-13T02:31:04.828 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542251 2026-02-13T02:31:04.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542251' 2026-02-13T02:31:04.828 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:31:04.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542252 -lt 60129542251 2026-02-13T02:31:04.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:31:04.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:31:04.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:31:05.138 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:31:05.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:31:05.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:31:05.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:31:05.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:31:05.150 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:31:05.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:31:05.312 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:31:05.312 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:31:05.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:31:05.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:05.313 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:31:05.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836596 2026-02-13T02:31:05.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836596 2026-02-13T02:31:05.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836596' 2026-02-13T02:31:05.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:05.386 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:31:05.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705777 2026-02-13T02:31:05.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705777 2026-02-13T02:31:05.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836596 1-38654705777' 2026-02-13T02:31:05.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:05.458 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:31:05.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542254 2026-02-13T02:31:05.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542254 2026-02-13T02:31:05.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836596 1-38654705777 2-60129542254' 2026-02-13T02:31:05.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:05.530 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836596 2026-02-13T02:31:05.531 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:05.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:31:05.532 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836596 2026-02-13T02:31:05.532 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:05.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836596 2026-02-13T02:31:05.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836596' 2026-02-13T02:31:05.534 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836596 2026-02-13T02:31:05.534 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:05.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836593 -lt 21474836596 2026-02-13T02:31:05.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:31:06.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:31:06.703 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:06.857 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836596 -lt 21474836596 2026-02-13T02:31:06.857 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:06.858 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705777 2026-02-13T02:31:06.858 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:06.859 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:31:06.860 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705777 2026-02-13T02:31:06.860 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:06.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705777 2026-02-13T02:31:06.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705777' 2026-02-13T02:31:06.862 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705777 2026-02-13T02:31:06.862 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:31:07.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705777 -lt 38654705777 2026-02-13T02:31:07.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:07.027 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542254 2026-02-13T02:31:07.027 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:07.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:31:07.029 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542254 2026-02-13T02:31:07.030 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:07.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542254 2026-02-13T02:31:07.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542254' 2026-02-13T02:31:07.031 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542254 2026-02-13T02:31:07.032 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:31:07.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542255 -lt 60129542254 2026-02-13T02:31:07.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:31:07.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:31:07.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:31:07.342 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:31:07.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:31:07.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:31:07.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:31:07.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:31:07.352 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:31:07.511 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:31:07.512 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:31:07.512 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:31:07.512 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:31:07.512 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:07.512 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:31:07.585 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836599 2026-02-13T02:31:07.585 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836599 2026-02-13T02:31:07.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836599' 2026-02-13T02:31:07.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:07.586 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:31:07.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705780 2026-02-13T02:31:07.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705780 2026-02-13T02:31:07.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836599 1-38654705780' 2026-02-13T02:31:07.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:07.658 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:31:07.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542258 2026-02-13T02:31:07.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542258 2026-02-13T02:31:07.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836599 1-38654705780 2-60129542258' 2026-02-13T02:31:07.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:07.732 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836599 2026-02-13T02:31:07.732 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:07.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:31:07.734 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836599 2026-02-13T02:31:07.734 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:07.735 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836599 2026-02-13T02:31:07.735 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836599' 2026-02-13T02:31:07.736 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836599 2026-02-13T02:31:07.736 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:07.899 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836599 -lt 21474836599 2026-02-13T02:31:07.899 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:07.900 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705780 2026-02-13T02:31:07.900 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:07.901 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:31:07.902 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705780 2026-02-13T02:31:07.902 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:07.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705780 2026-02-13T02:31:07.903 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705780 2026-02-13T02:31:07.904 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705780' 2026-02-13T02:31:07.904 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:31:08.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705780 -lt 38654705780 2026-02-13T02:31:08.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:08.068 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542258 2026-02-13T02:31:08.068 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:08.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:31:08.070 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542258 2026-02-13T02:31:08.070 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:08.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542258 2026-02-13T02:31:08.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542258' 2026-02-13T02:31:08.072 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542258 2026-02-13T02:31:08.072 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:31:08.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542257 -lt 60129542258 2026-02-13T02:31:08.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:31:09.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:31:09.239 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:31:09.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542257 -lt 60129542258 2026-02-13T02:31:09.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:31:10.397 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:31:10.398 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:31:10.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542260 -lt 60129542258 2026-02-13T02:31:10.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:31:10.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:31:10.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:31:10.712 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:31:10.723 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:31:10.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:31:10.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:31:10.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:31:10.724 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:31:10.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:31:10.884 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:31:10.884 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:31:10.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:31:10.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:10.885 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:31:10.959 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836603 2026-02-13T02:31:10.959 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836603 2026-02-13T02:31:10.959 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836603' 2026-02-13T02:31:10.959 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:10.959 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:31:11.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705784 2026-02-13T02:31:11.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705784 2026-02-13T02:31:11.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836603 1-38654705784' 2026-02-13T02:31:11.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:11.029 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:31:11.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542262 2026-02-13T02:31:11.104 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542262 2026-02-13T02:31:11.104 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836603 1-38654705784 2-60129542262' 2026-02-13T02:31:11.104 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:11.104 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836603 2026-02-13T02:31:11.104 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:11.105 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:31:11.106 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836603 2026-02-13T02:31:11.106 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:11.107 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836603 2026-02-13T02:31:11.108 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836603 2026-02-13T02:31:11.108 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836603' 2026-02-13T02:31:11.108 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:11.269 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836601 -lt 21474836603 2026-02-13T02:31:11.269 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:31:12.270 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:31:12.271 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:12.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836604 -lt 21474836603 2026-02-13T02:31:12.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:12.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705784 2026-02-13T02:31:12.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:12.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:31:12.435 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705784 2026-02-13T02:31:12.435 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:12.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705784 2026-02-13T02:31:12.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705784' 2026-02-13T02:31:12.436 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705784 2026-02-13T02:31:12.437 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:31:12.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705785 -lt 38654705784 2026-02-13T02:31:12.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:12.607 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542262 2026-02-13T02:31:12.607 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:12.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:31:12.609 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542262 2026-02-13T02:31:12.609 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:12.611 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542262 2026-02-13T02:31:12.611 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542262 2026-02-13T02:31:12.611 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542262' 2026-02-13T02:31:12.611 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:31:12.769 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542263 -lt 60129542262 2026-02-13T02:31:12.770 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:31:12.770 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:31:12.770 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:31:12.922 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:31:12.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:31:12.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:31:12.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:31:12.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:31:12.932 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:31:13.089 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:31:13.089 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:31:13.089 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:31:13.089 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:31:13.089 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:13.089 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:31:13.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836607 2026-02-13T02:31:13.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836607 2026-02-13T02:31:13.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836607' 2026-02-13T02:31:13.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:13.164 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:31:13.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705788 2026-02-13T02:31:13.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705788 2026-02-13T02:31:13.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836607 1-38654705788' 2026-02-13T02:31:13.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:13.237 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:31:13.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542265 2026-02-13T02:31:13.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542265 2026-02-13T02:31:13.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836607 1-38654705788 2-60129542265' 2026-02-13T02:31:13.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:13.311 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836607 2026-02-13T02:31:13.311 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:13.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:31:13.313 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836607 2026-02-13T02:31:13.313 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:13.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836607 2026-02-13T02:31:13.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836607' 2026-02-13T02:31:13.315 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836607 2026-02-13T02:31:13.315 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:13.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836604 -lt 21474836607 2026-02-13T02:31:13.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:31:14.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:31:14.486 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:14.645 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836607 -lt 21474836607 2026-02-13T02:31:14.645 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:14.646 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705788 2026-02-13T02:31:14.646 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:14.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:31:14.648 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705788 2026-02-13T02:31:14.648 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:14.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705788 2026-02-13T02:31:14.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705788' 2026-02-13T02:31:14.650 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705788 2026-02-13T02:31:14.650 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:31:14.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705788 -lt 38654705788 2026-02-13T02:31:14.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:14.822 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542265 2026-02-13T02:31:14.822 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:14.823 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:31:14.824 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542265 2026-02-13T02:31:14.824 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:14.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542265 2026-02-13T02:31:14.825 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542265 2026-02-13T02:31:14.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542265' 2026-02-13T02:31:14.826 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:31:14.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542266 -lt 60129542265 2026-02-13T02:31:14.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:31:14.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:31:14.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:31:15.147 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:31:15.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:31:15.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:31:15.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:31:15.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:31:15.157 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:31:15.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:31:15.313 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:31:15.313 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:31:15.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:31:15.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:15.314 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:31:15.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836610 2026-02-13T02:31:15.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836610 2026-02-13T02:31:15.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836610' 2026-02-13T02:31:15.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:15.387 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:31:15.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705791 2026-02-13T02:31:15.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705791 2026-02-13T02:31:15.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836610 1-38654705791' 2026-02-13T02:31:15.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:15.461 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:31:15.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542268 2026-02-13T02:31:15.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542268 2026-02-13T02:31:15.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836610 1-38654705791 2-60129542268' 2026-02-13T02:31:15.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:15.532 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836610 2026-02-13T02:31:15.532 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:15.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:31:15.533 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836610 2026-02-13T02:31:15.533 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:15.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836610 2026-02-13T02:31:15.535 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836610 2026-02-13T02:31:15.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836610' 2026-02-13T02:31:15.535 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:15.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836607 -lt 21474836610 2026-02-13T02:31:15.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:31:16.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:31:16.699 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:16.857 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836610 -lt 21474836610 2026-02-13T02:31:16.857 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:16.858 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705791 2026-02-13T02:31:16.858 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:16.859 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:31:16.860 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705791 2026-02-13T02:31:16.860 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:16.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705791 2026-02-13T02:31:16.862 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705791 2026-02-13T02:31:16.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705791' 2026-02-13T02:31:16.862 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:31:17.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705791 -lt 38654705791 2026-02-13T02:31:17.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:17.037 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542268 2026-02-13T02:31:17.037 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:17.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:31:17.039 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542268 2026-02-13T02:31:17.039 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:17.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542268 2026-02-13T02:31:17.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542268' 2026-02-13T02:31:17.040 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542268 2026-02-13T02:31:17.041 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:31:17.202 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542269 -lt 60129542268 2026-02-13T02:31:17.203 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:31:17.203 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:31:17.203 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:31:17.353 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:31:17.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:31:17.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:31:17.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:31:17.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:31:17.366 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:31:17.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:31:17.526 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:31:17.527 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:31:17.527 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:31:17.527 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:17.527 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:31:17.600 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836613 2026-02-13T02:31:17.600 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836613 2026-02-13T02:31:17.600 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836613' 2026-02-13T02:31:17.600 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:17.600 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:31:17.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705794 2026-02-13T02:31:17.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705794 2026-02-13T02:31:17.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836613 1-38654705794' 2026-02-13T02:31:17.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:17.670 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:31:17.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542272 2026-02-13T02:31:17.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542272 2026-02-13T02:31:17.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836613 1-38654705794 2-60129542272' 2026-02-13T02:31:17.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:17.742 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836613 2026-02-13T02:31:17.742 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:17.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:31:17.744 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836613 2026-02-13T02:31:17.744 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:17.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836613 2026-02-13T02:31:17.746 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836613 2026-02-13T02:31:17.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836613' 2026-02-13T02:31:17.747 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:17.914 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836613 -lt 21474836613 2026-02-13T02:31:17.914 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:17.914 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705794 2026-02-13T02:31:17.915 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:17.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:31:17.916 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705794 2026-02-13T02:31:17.916 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:17.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705794 2026-02-13T02:31:17.918 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705794 2026-02-13T02:31:17.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705794' 2026-02-13T02:31:17.919 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:31:18.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705794 -lt 38654705794 2026-02-13T02:31:18.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:18.076 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542272 2026-02-13T02:31:18.076 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:18.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:31:18.078 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542272 2026-02-13T02:31:18.078 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:18.079 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542272 2026-02-13T02:31:18.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542272' 2026-02-13T02:31:18.080 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542272 2026-02-13T02:31:18.080 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:31:18.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542271 -lt 60129542272 2026-02-13T02:31:18.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:31:19.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:31:19.245 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:31:19.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542271 -lt 60129542272 2026-02-13T02:31:19.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:31:20.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:31:20.409 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:31:20.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542274 -lt 60129542272 2026-02-13T02:31:20.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:31:20.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:31:20.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:31:20.715 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:31:20.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:31:20.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:31:20.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:31:20.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:31:20.725 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:31:20.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:31:20.883 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:31:20.883 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:31:20.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:31:20.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:20.883 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:31:20.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836617 2026-02-13T02:31:20.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836617 2026-02-13T02:31:20.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836617' 2026-02-13T02:31:20.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:20.957 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:31:21.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705798 2026-02-13T02:31:21.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705798 2026-02-13T02:31:21.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836617 1-38654705798' 2026-02-13T02:31:21.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:21.028 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:31:21.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542276 2026-02-13T02:31:21.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542276 2026-02-13T02:31:21.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836617 1-38654705798 2-60129542276' 2026-02-13T02:31:21.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:21.103 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836617 2026-02-13T02:31:21.103 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:21.104 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:31:21.105 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836617 2026-02-13T02:31:21.105 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:21.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836617 2026-02-13T02:31:21.107 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836617 2026-02-13T02:31:21.107 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836617' 2026-02-13T02:31:21.107 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:21.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836615 -lt 21474836617 2026-02-13T02:31:21.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:31:22.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:31:22.276 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:22.439 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836618 -lt 21474836617 2026-02-13T02:31:22.439 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:22.440 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705798 2026-02-13T02:31:22.440 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:22.441 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:31:22.442 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705798 2026-02-13T02:31:22.442 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:22.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705798 2026-02-13T02:31:22.444 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705798 2026-02-13T02:31:22.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705798' 2026-02-13T02:31:22.444 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:31:22.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705799 -lt 38654705798 2026-02-13T02:31:22.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:22.605 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542276 2026-02-13T02:31:22.605 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:22.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:31:22.607 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542276 2026-02-13T02:31:22.607 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:22.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542276 2026-02-13T02:31:22.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542276' 2026-02-13T02:31:22.609 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542276 2026-02-13T02:31:22.610 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:31:22.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542277 -lt 60129542276 2026-02-13T02:31:22.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:31:22.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:31:22.779 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:31:22.930 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:31:22.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:31:22.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:31:22.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:31:22.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:31:22.941 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:31:23.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:31:23.100 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:31:23.100 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:31:23.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:31:23.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:23.101 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:31:23.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836621 2026-02-13T02:31:23.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836621 2026-02-13T02:31:23.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836621' 2026-02-13T02:31:23.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:23.174 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:31:23.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705802 2026-02-13T02:31:23.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705802 2026-02-13T02:31:23.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836621 1-38654705802' 2026-02-13T02:31:23.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:23.248 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:31:23.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542279 2026-02-13T02:31:23.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542279 2026-02-13T02:31:23.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836621 1-38654705802 2-60129542279' 2026-02-13T02:31:23.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:23.321 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836621 2026-02-13T02:31:23.321 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:23.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:31:23.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836621 2026-02-13T02:31:23.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:23.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836621 2026-02-13T02:31:23.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836621' 2026-02-13T02:31:23.325 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836621 2026-02-13T02:31:23.325 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:23.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836618 -lt 21474836621 2026-02-13T02:31:23.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:31:24.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:31:24.490 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:24.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836621 -lt 21474836621 2026-02-13T02:31:24.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:24.655 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705802 2026-02-13T02:31:24.655 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:24.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:31:24.656 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705802 2026-02-13T02:31:24.657 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:24.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705802 2026-02-13T02:31:24.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705802' 2026-02-13T02:31:24.659 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705802 2026-02-13T02:31:24.659 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:31:24.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705802 -lt 38654705802 2026-02-13T02:31:24.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:24.821 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542279 2026-02-13T02:31:24.821 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:24.823 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:31:24.823 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542279 2026-02-13T02:31:24.823 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:24.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542279 2026-02-13T02:31:24.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542279' 2026-02-13T02:31:24.825 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542279 2026-02-13T02:31:24.826 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:31:24.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542280 -lt 60129542279 2026-02-13T02:31:25.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:31:25.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:31:25.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:31:25.151 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:31:25.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:31:25.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:31:25.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:31:25.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:31:25.164 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:31:25.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:31:25.324 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:31:25.324 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:31:25.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:31:25.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:25.325 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:31:25.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836624 2026-02-13T02:31:25.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836624 2026-02-13T02:31:25.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836624' 2026-02-13T02:31:25.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:25.400 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:31:25.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705805 2026-02-13T02:31:25.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705805 2026-02-13T02:31:25.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836624 1-38654705805' 2026-02-13T02:31:25.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:25.471 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:31:25.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542282 2026-02-13T02:31:25.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542282 2026-02-13T02:31:25.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836624 1-38654705805 2-60129542282' 2026-02-13T02:31:25.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:25.542 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836624 2026-02-13T02:31:25.542 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:25.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:31:25.544 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836624 2026-02-13T02:31:25.544 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:25.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836624 2026-02-13T02:31:25.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836624' 2026-02-13T02:31:25.545 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836624 2026-02-13T02:31:25.546 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:25.705 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836621 -lt 21474836624 2026-02-13T02:31:25.705 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:31:26.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:31:26.707 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:26.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836624 -lt 21474836624 2026-02-13T02:31:26.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:26.865 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705805 2026-02-13T02:31:26.866 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:26.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:31:26.868 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705805 2026-02-13T02:31:26.868 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:26.869 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705805 2026-02-13T02:31:26.869 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705805 2026-02-13T02:31:26.869 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705805' 2026-02-13T02:31:26.870 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:31:27.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705805 -lt 38654705805 2026-02-13T02:31:27.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:27.030 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542282 2026-02-13T02:31:27.030 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:27.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:31:27.032 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542282 2026-02-13T02:31:27.032 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:27.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542282 2026-02-13T02:31:27.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542282' 2026-02-13T02:31:27.034 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542282 2026-02-13T02:31:27.034 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:31:27.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542283 -lt 60129542282 2026-02-13T02:31:27.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:31:27.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:31:27.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:31:27.354 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:31:27.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:31:27.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:31:27.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:31:27.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:31:27.364 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:31:27.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:31:27.522 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:31:27.522 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:31:27.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:31:27.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:27.522 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:31:27.594 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836627 2026-02-13T02:31:27.594 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836627 2026-02-13T02:31:27.594 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836627' 2026-02-13T02:31:27.594 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:27.594 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:31:27.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705808 2026-02-13T02:31:27.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705808 2026-02-13T02:31:27.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836627 1-38654705808' 2026-02-13T02:31:27.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:27.665 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:31:27.737 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542286 2026-02-13T02:31:27.737 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542286 2026-02-13T02:31:27.737 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836627 1-38654705808 2-60129542286' 2026-02-13T02:31:27.737 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:27.738 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836627 2026-02-13T02:31:27.738 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:27.739 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:31:27.740 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836627 2026-02-13T02:31:27.740 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:27.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836627 2026-02-13T02:31:27.742 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836627 2026-02-13T02:31:27.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836627' 2026-02-13T02:31:27.742 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:27.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836627 -lt 21474836627 2026-02-13T02:31:27.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:27.908 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705808 2026-02-13T02:31:27.908 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:27.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:31:27.910 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705808 2026-02-13T02:31:27.910 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:27.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705808 2026-02-13T02:31:27.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705808' 2026-02-13T02:31:27.912 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705808 2026-02-13T02:31:27.912 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:31:28.066 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705808 -lt 38654705808 2026-02-13T02:31:28.066 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:28.067 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542286 2026-02-13T02:31:28.067 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:28.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:31:28.068 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542286 2026-02-13T02:31:28.068 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:28.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542286 2026-02-13T02:31:28.070 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542286 2026-02-13T02:31:28.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542286' 2026-02-13T02:31:28.071 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:31:28.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542285 -lt 60129542286 2026-02-13T02:31:28.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:31:29.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:31:29.239 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:31:29.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542285 -lt 60129542286 2026-02-13T02:31:29.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:31:30.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:31:30.401 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:31:30.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542288 -lt 60129542286 2026-02-13T02:31:30.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:31:30.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:31:30.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:31:30.712 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:31:30.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:31:30.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:31:30.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:31:30.723 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:31:30.723 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:31:30.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:31:30.881 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:31:30.881 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:31:30.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:31:30.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:30.881 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:31:30.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836631 2026-02-13T02:31:30.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836631 2026-02-13T02:31:30.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836631' 2026-02-13T02:31:30.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:30.954 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:31:31.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705812 2026-02-13T02:31:31.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705812 2026-02-13T02:31:31.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836631 1-38654705812' 2026-02-13T02:31:31.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:31.026 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:31:31.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542290 2026-02-13T02:31:31.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542290 2026-02-13T02:31:31.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836631 1-38654705812 2-60129542290' 2026-02-13T02:31:31.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:31.101 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836631 2026-02-13T02:31:31.101 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:31.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:31:31.103 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836631 2026-02-13T02:31:31.103 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:31.104 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836631 2026-02-13T02:31:31.104 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836631' 2026-02-13T02:31:31.104 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836631 2026-02-13T02:31:31.105 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:31.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836629 -lt 21474836631 2026-02-13T02:31:31.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:31:32.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:31:32.266 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:32.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836632 -lt 21474836631 2026-02-13T02:31:32.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:32.428 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705812 2026-02-13T02:31:32.429 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:32.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:31:32.431 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705812 2026-02-13T02:31:32.431 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:32.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705812 2026-02-13T02:31:32.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705812' 2026-02-13T02:31:32.432 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705812 2026-02-13T02:31:32.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:31:32.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705813 -lt 38654705812 2026-02-13T02:31:32.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:32.603 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542290 2026-02-13T02:31:32.603 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:32.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:31:32.605 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542290 2026-02-13T02:31:32.605 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:32.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542290 2026-02-13T02:31:32.607 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542290 2026-02-13T02:31:32.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542290' 2026-02-13T02:31:32.607 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:31:32.769 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542291 -lt 60129542290 2026-02-13T02:31:32.769 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:31:32.769 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:31:32.769 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:31:32.922 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:31:32.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:31:32.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:31:32.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:31:32.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:31:32.932 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:31:33.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:31:33.093 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:31:33.093 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:31:33.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:31:33.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:33.094 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:31:33.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836635 2026-02-13T02:31:33.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836635 2026-02-13T02:31:33.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836635' 2026-02-13T02:31:33.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:33.166 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:31:33.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705816 2026-02-13T02:31:33.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705816 2026-02-13T02:31:33.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836635 1-38654705816' 2026-02-13T02:31:33.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:33.237 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:31:33.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542293 2026-02-13T02:31:33.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542293 2026-02-13T02:31:33.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836635 1-38654705816 2-60129542293' 2026-02-13T02:31:33.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:33.313 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836635 2026-02-13T02:31:33.313 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:33.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:31:33.314 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836635 2026-02-13T02:31:33.314 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:33.315 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836635 2026-02-13T02:31:33.315 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836635' 2026-02-13T02:31:33.316 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836635 2026-02-13T02:31:33.316 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:33.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836632 -lt 21474836635 2026-02-13T02:31:33.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:31:34.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:31:34.482 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:34.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836635 -lt 21474836635 2026-02-13T02:31:34.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:34.641 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705816 2026-02-13T02:31:34.641 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:34.642 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:31:34.643 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705816 2026-02-13T02:31:34.643 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:34.644 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705816 2026-02-13T02:31:34.644 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705816' 2026-02-13T02:31:34.645 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705816 2026-02-13T02:31:34.645 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:31:34.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705816 -lt 38654705816 2026-02-13T02:31:34.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:34.819 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542293 2026-02-13T02:31:34.819 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:34.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:31:34.820 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542293 2026-02-13T02:31:34.820 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:34.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542293 2026-02-13T02:31:34.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542293' 2026-02-13T02:31:34.822 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542293 2026-02-13T02:31:34.822 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:31:34.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542294 -lt 60129542293 2026-02-13T02:31:34.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:31:34.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:31:34.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:31:35.128 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:31:35.138 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:31:35.138 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:31:35.138 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:31:35.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:31:35.139 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:31:35.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:31:35.306 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:31:35.306 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:31:35.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:31:35.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:35.306 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:31:35.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836638 2026-02-13T02:31:35.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836638 2026-02-13T02:31:35.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836638' 2026-02-13T02:31:35.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:35.379 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:31:35.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705819 2026-02-13T02:31:35.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705819 2026-02-13T02:31:35.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836638 1-38654705819' 2026-02-13T02:31:35.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:35.452 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:31:35.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542296 2026-02-13T02:31:35.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542296 2026-02-13T02:31:35.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836638 1-38654705819 2-60129542296' 2026-02-13T02:31:35.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:35.527 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836638 2026-02-13T02:31:35.527 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:35.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:31:35.529 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836638 2026-02-13T02:31:35.529 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:35.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836638 2026-02-13T02:31:35.531 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836638 2026-02-13T02:31:35.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836638' 2026-02-13T02:31:35.531 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:35.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836635 -lt 21474836638 2026-02-13T02:31:35.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:31:36.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:31:36.690 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:36.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836638 -lt 21474836638 2026-02-13T02:31:36.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:36.850 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705819 2026-02-13T02:31:36.850 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:36.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:31:36.852 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705819 2026-02-13T02:31:36.852 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:36.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705819 2026-02-13T02:31:36.854 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705819 2026-02-13T02:31:36.854 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705819' 2026-02-13T02:31:36.855 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:31:37.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705819 -lt 38654705819 2026-02-13T02:31:37.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:37.019 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542296 2026-02-13T02:31:37.019 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:37.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:31:37.021 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542296 2026-02-13T02:31:37.021 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:37.023 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542296 2026-02-13T02:31:37.023 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542296' 2026-02-13T02:31:37.023 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542296 2026-02-13T02:31:37.023 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:31:37.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542297 -lt 60129542296 2026-02-13T02:31:37.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:31:37.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:31:37.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:31:37.332 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:31:37.343 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:31:37.343 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:31:37.343 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:31:37.343 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:31:37.343 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:31:37.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:31:37.503 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:31:37.503 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:31:37.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:31:37.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:37.503 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:31:37.575 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836641 2026-02-13T02:31:37.575 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836641 2026-02-13T02:31:37.575 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836641' 2026-02-13T02:31:37.575 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:37.575 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:31:37.646 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705822 2026-02-13T02:31:37.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705822 2026-02-13T02:31:37.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836641 1-38654705822' 2026-02-13T02:31:37.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:37.647 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:31:37.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542300 2026-02-13T02:31:37.718 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542300 2026-02-13T02:31:37.718 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836641 1-38654705822 2-60129542300' 2026-02-13T02:31:37.718 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:37.718 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836641 2026-02-13T02:31:37.718 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:37.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:31:37.720 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836641 2026-02-13T02:31:37.720 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:37.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836641 2026-02-13T02:31:37.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836641' 2026-02-13T02:31:37.722 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836641 2026-02-13T02:31:37.722 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:37.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836641 -lt 21474836641 2026-02-13T02:31:37.886 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:37.886 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705822 2026-02-13T02:31:37.886 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:37.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:31:37.888 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705822 2026-02-13T02:31:37.888 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:37.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705822 2026-02-13T02:31:37.890 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705822 2026-02-13T02:31:37.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705822' 2026-02-13T02:31:37.890 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:31:38.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705822 -lt 38654705822 2026-02-13T02:31:38.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:38.063 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542300 2026-02-13T02:31:38.063 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:38.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:31:38.064 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542300 2026-02-13T02:31:38.064 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:38.066 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542300 2026-02-13T02:31:38.066 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542300 2026-02-13T02:31:38.066 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542300' 2026-02-13T02:31:38.067 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:31:38.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542300 -lt 60129542300 2026-02-13T02:31:38.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:31:38.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:31:38.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:31:38.389 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:31:38.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:31:38.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:31:38.401 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:31:38.401 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:31:38.401 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:31:38.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:31:38.560 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:31:38.560 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:31:38.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:31:38.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:38.560 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:31:38.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836643 2026-02-13T02:31:38.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836643 2026-02-13T02:31:38.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836643' 2026-02-13T02:31:38.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:38.633 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:31:38.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705824 2026-02-13T02:31:38.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705824 2026-02-13T02:31:38.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836643 1-38654705824' 2026-02-13T02:31:38.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:38.708 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:31:38.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542302 2026-02-13T02:31:38.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542302 2026-02-13T02:31:38.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836643 1-38654705824 2-60129542302' 2026-02-13T02:31:38.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:38.779 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836643 2026-02-13T02:31:38.779 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:38.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:31:38.780 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836643 2026-02-13T02:31:38.780 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:38.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836643 2026-02-13T02:31:38.782 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836643 2026-02-13T02:31:38.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836643' 2026-02-13T02:31:38.783 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:38.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836641 -lt 21474836643 2026-02-13T02:31:38.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:31:39.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:31:39.948 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:40.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836644 -lt 21474836643 2026-02-13T02:31:40.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:40.115 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705824 2026-02-13T02:31:40.115 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:40.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:31:40.117 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705824 2026-02-13T02:31:40.118 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:40.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705824 2026-02-13T02:31:40.119 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705824 2026-02-13T02:31:40.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705824' 2026-02-13T02:31:40.119 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:31:40.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705825 -lt 38654705824 2026-02-13T02:31:40.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:40.279 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542302 2026-02-13T02:31:40.279 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:40.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:31:40.280 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542302 2026-02-13T02:31:40.280 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:40.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542302 2026-02-13T02:31:40.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542302' 2026-02-13T02:31:40.282 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542302 2026-02-13T02:31:40.283 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:31:40.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542303 -lt 60129542302 2026-02-13T02:31:40.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:31:40.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:31:40.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:31:40.599 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:31:40.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:31:40.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:31:40.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:31:40.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:31:40.609 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:31:40.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:31:40.772 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:31:40.772 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:31:40.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:31:40.773 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:40.773 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:31:40.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836646 2026-02-13T02:31:40.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836646 2026-02-13T02:31:40.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836646' 2026-02-13T02:31:40.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:40.842 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:31:40.914 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705827 2026-02-13T02:31:40.914 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705827 2026-02-13T02:31:40.914 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836646 1-38654705827' 2026-02-13T02:31:40.914 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:40.914 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:31:40.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542305 2026-02-13T02:31:40.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542305 2026-02-13T02:31:40.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836646 1-38654705827 2-60129542305' 2026-02-13T02:31:40.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:40.986 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836646 2026-02-13T02:31:40.986 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:40.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:31:40.988 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836646 2026-02-13T02:31:40.988 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:40.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836646 2026-02-13T02:31:40.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836646' 2026-02-13T02:31:40.990 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836646 2026-02-13T02:31:40.990 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:41.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836644 -lt 21474836646 2026-02-13T02:31:41.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:31:42.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:31:42.160 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:42.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836647 -lt 21474836646 2026-02-13T02:31:42.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:42.315 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705827 2026-02-13T02:31:42.315 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:42.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:31:42.317 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705827 2026-02-13T02:31:42.317 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:42.319 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705827 2026-02-13T02:31:42.319 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705827' 2026-02-13T02:31:42.319 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705827 2026-02-13T02:31:42.319 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:31:42.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705828 -lt 38654705827 2026-02-13T02:31:42.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:42.481 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542305 2026-02-13T02:31:42.481 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:42.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:31:42.482 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542305 2026-02-13T02:31:42.482 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:42.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542305 2026-02-13T02:31:42.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542305' 2026-02-13T02:31:42.484 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542305 2026-02-13T02:31:42.485 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:31:42.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542306 -lt 60129542305 2026-02-13T02:31:42.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:31:42.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:31:42.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:31:42.797 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:31:42.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:31:42.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:31:42.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:31:42.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:31:42.809 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:31:42.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:31:42.974 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:31:42.974 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:31:42.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:31:42.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:42.975 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:31:43.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836649 2026-02-13T02:31:43.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836649 2026-02-13T02:31:43.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836649' 2026-02-13T02:31:43.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:43.045 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:31:43.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705830 2026-02-13T02:31:43.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705830 2026-02-13T02:31:43.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836649 1-38654705830' 2026-02-13T02:31:43.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:43.119 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:31:43.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542308 2026-02-13T02:31:43.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542308 2026-02-13T02:31:43.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836649 1-38654705830 2-60129542308' 2026-02-13T02:31:43.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:43.194 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836649 2026-02-13T02:31:43.194 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:43.195 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:31:43.196 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836649 2026-02-13T02:31:43.196 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:43.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836649 2026-02-13T02:31:43.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836649' 2026-02-13T02:31:43.198 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836649 2026-02-13T02:31:43.198 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:43.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836647 -lt 21474836649 2026-02-13T02:31:43.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:31:44.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:31:44.366 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:44.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836650 -lt 21474836649 2026-02-13T02:31:44.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:44.531 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705830 2026-02-13T02:31:44.531 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:44.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:31:44.533 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705830 2026-02-13T02:31:44.533 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:44.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705830 2026-02-13T02:31:44.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705830' 2026-02-13T02:31:44.535 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705830 2026-02-13T02:31:44.535 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:31:44.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705831 -lt 38654705830 2026-02-13T02:31:44.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:44.707 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542308 2026-02-13T02:31:44.707 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:44.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:31:44.708 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542308 2026-02-13T02:31:44.709 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:44.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542308 2026-02-13T02:31:44.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542308' 2026-02-13T02:31:44.710 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542308 2026-02-13T02:31:44.710 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:31:44.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542309 -lt 60129542308 2026-02-13T02:31:44.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:31:44.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:31:44.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:31:45.025 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:31:45.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:31:45.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:31:45.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:31:45.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:31:45.035 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:31:45.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:31:45.197 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:31:45.198 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:31:45.198 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:31:45.198 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:45.198 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:31:45.271 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836653 2026-02-13T02:31:45.271 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836653 2026-02-13T02:31:45.271 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836653' 2026-02-13T02:31:45.271 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:45.271 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:31:45.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705834 2026-02-13T02:31:45.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705834 2026-02-13T02:31:45.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836653 1-38654705834' 2026-02-13T02:31:45.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:45.342 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:31:45.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542311 2026-02-13T02:31:45.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542311 2026-02-13T02:31:45.416 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836653 1-38654705834 2-60129542311' 2026-02-13T02:31:45.416 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:45.416 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836653 2026-02-13T02:31:45.416 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:45.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:31:45.418 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836653 2026-02-13T02:31:45.418 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:45.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836653 2026-02-13T02:31:45.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836653' 2026-02-13T02:31:45.420 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836653 2026-02-13T02:31:45.420 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:45.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836650 -lt 21474836653 2026-02-13T02:31:45.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:31:46.580 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:31:46.580 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:46.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836653 -lt 21474836653 2026-02-13T02:31:46.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:46.743 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705834 2026-02-13T02:31:46.743 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:46.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:31:46.745 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705834 2026-02-13T02:31:46.745 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:46.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705834 2026-02-13T02:31:46.747 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705834 2026-02-13T02:31:46.747 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705834' 2026-02-13T02:31:46.747 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:31:46.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705834 -lt 38654705834 2026-02-13T02:31:46.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:46.919 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542311 2026-02-13T02:31:46.919 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:46.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:31:46.921 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542311 2026-02-13T02:31:46.921 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:46.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542311 2026-02-13T02:31:46.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542311' 2026-02-13T02:31:46.923 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542311 2026-02-13T02:31:46.923 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:31:47.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542312 -lt 60129542311 2026-02-13T02:31:47.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:31:47.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:31:47.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:31:47.241 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:31:47.251 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:31:47.251 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:31:47.251 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:31:47.251 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:31:47.251 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:31:47.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:31:47.415 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:31:47.415 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:31:47.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:31:47.416 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:47.416 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:31:47.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836656 2026-02-13T02:31:47.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836656 2026-02-13T02:31:47.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836656' 2026-02-13T02:31:47.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:47.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:31:47.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705837 2026-02-13T02:31:47.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705837 2026-02-13T02:31:47.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836656 1-38654705837' 2026-02-13T02:31:47.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:47.562 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:31:47.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542314 2026-02-13T02:31:47.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542314 2026-02-13T02:31:47.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836656 1-38654705837 2-60129542314' 2026-02-13T02:31:47.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:47.634 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836656 2026-02-13T02:31:47.634 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:47.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:31:47.636 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836656 2026-02-13T02:31:47.636 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:47.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836656 2026-02-13T02:31:47.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836656' 2026-02-13T02:31:47.638 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836656 2026-02-13T02:31:47.638 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:47.806 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836656 -lt 21474836656 2026-02-13T02:31:47.806 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:47.807 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705837 2026-02-13T02:31:47.807 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:47.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:31:47.808 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705837 2026-02-13T02:31:47.809 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:47.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705837 2026-02-13T02:31:47.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705837' 2026-02-13T02:31:47.811 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705837 2026-02-13T02:31:47.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:31:47.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705837 -lt 38654705837 2026-02-13T02:31:47.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:47.969 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542314 2026-02-13T02:31:47.969 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:47.970 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:31:47.971 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542314 2026-02-13T02:31:47.971 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:47.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542314 2026-02-13T02:31:47.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542314' 2026-02-13T02:31:47.973 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542314 2026-02-13T02:31:47.974 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:31:48.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542315 -lt 60129542314 2026-02-13T02:31:48.144 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:31:48.144 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:31:48.144 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:31:48.291 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:31:48.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:31:48.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:31:48.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:31:48.300 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:31:48.300 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:31:48.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:31:48.465 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:31:48.465 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:31:48.465 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:31:48.465 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:48.465 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:31:48.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836658 2026-02-13T02:31:48.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836658 2026-02-13T02:31:48.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836658' 2026-02-13T02:31:48.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:48.539 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:31:48.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705839 2026-02-13T02:31:48.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705839 2026-02-13T02:31:48.613 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836658 1-38654705839' 2026-02-13T02:31:48.613 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:48.613 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:31:48.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542317 2026-02-13T02:31:48.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542317 2026-02-13T02:31:48.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836658 1-38654705839 2-60129542317' 2026-02-13T02:31:48.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:48.685 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836658 2026-02-13T02:31:48.686 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:48.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:31:48.687 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836658 2026-02-13T02:31:48.687 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:48.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836658 2026-02-13T02:31:48.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836658' 2026-02-13T02:31:48.689 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836658 2026-02-13T02:31:48.689 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:48.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836656 -lt 21474836658 2026-02-13T02:31:48.859 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:31:49.859 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:31:49.860 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:50.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836659 -lt 21474836658 2026-02-13T02:31:50.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:50.020 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705839 2026-02-13T02:31:50.020 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:50.021 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:31:50.021 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705839 2026-02-13T02:31:50.021 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:50.023 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705839 2026-02-13T02:31:50.023 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705839' 2026-02-13T02:31:50.023 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705839 2026-02-13T02:31:50.024 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:31:50.183 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705840 -lt 38654705839 2026-02-13T02:31:50.184 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:50.184 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542317 2026-02-13T02:31:50.185 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:50.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:31:50.186 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542317 2026-02-13T02:31:50.186 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:50.187 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542317 2026-02-13T02:31:50.188 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542317 2026-02-13T02:31:50.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542317' 2026-02-13T02:31:50.189 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:31:50.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542318 -lt 60129542317 2026-02-13T02:31:50.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:31:50.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:31:50.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:31:50.499 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:31:50.508 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:31:50.508 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:31:50.508 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:31:50.508 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:31:50.509 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:31:50.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:31:50.664 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:31:50.664 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:31:50.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:31:50.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:50.665 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:31:50.739 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836661 2026-02-13T02:31:50.739 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836661 2026-02-13T02:31:50.739 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836661' 2026-02-13T02:31:50.739 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:50.739 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:31:50.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705842 2026-02-13T02:31:50.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705842 2026-02-13T02:31:50.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836661 1-38654705842' 2026-02-13T02:31:50.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:50.814 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:31:50.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542320 2026-02-13T02:31:50.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542320 2026-02-13T02:31:50.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836661 1-38654705842 2-60129542320' 2026-02-13T02:31:50.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:50.888 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836661 2026-02-13T02:31:50.888 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:50.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:31:50.890 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836661 2026-02-13T02:31:50.890 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:50.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836661 2026-02-13T02:31:50.892 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836661 2026-02-13T02:31:50.892 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836661' 2026-02-13T02:31:50.892 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:51.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836659 -lt 21474836661 2026-02-13T02:31:51.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:31:52.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:31:52.062 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:52.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836662 -lt 21474836661 2026-02-13T02:31:52.222 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:52.222 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705842 2026-02-13T02:31:52.222 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:52.223 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:31:52.224 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705842 2026-02-13T02:31:52.224 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:52.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705842 2026-02-13T02:31:52.226 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705842 2026-02-13T02:31:52.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705842' 2026-02-13T02:31:52.226 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:31:52.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705843 -lt 38654705842 2026-02-13T02:31:52.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:52.401 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542320 2026-02-13T02:31:52.401 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:52.402 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:31:52.402 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542320 2026-02-13T02:31:52.403 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:52.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542320 2026-02-13T02:31:52.404 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542320 2026-02-13T02:31:52.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542320' 2026-02-13T02:31:52.405 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:31:52.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542321 -lt 60129542320 2026-02-13T02:31:52.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:31:52.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:31:52.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:31:52.722 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:31:52.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:31:52.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:31:52.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:31:52.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:31:52.733 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:31:52.892 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:31:52.892 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:31:52.892 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:31:52.893 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:31:52.893 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:52.893 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:31:52.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836664 2026-02-13T02:31:52.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836664 2026-02-13T02:31:52.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836664' 2026-02-13T02:31:52.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:52.969 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:31:53.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705845 2026-02-13T02:31:53.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705845 2026-02-13T02:31:53.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836664 1-38654705845' 2026-02-13T02:31:53.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:53.039 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:31:53.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542323 2026-02-13T02:31:53.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542323 2026-02-13T02:31:53.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836664 1-38654705845 2-60129542323' 2026-02-13T02:31:53.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:53.111 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836664 2026-02-13T02:31:53.111 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:53.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:31:53.112 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836664 2026-02-13T02:31:53.112 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:53.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836664 2026-02-13T02:31:53.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836664' 2026-02-13T02:31:53.113 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836664 2026-02-13T02:31:53.114 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:53.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836662 -lt 21474836664 2026-02-13T02:31:53.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:31:54.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:31:54.283 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:54.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836665 -lt 21474836664 2026-02-13T02:31:54.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:54.447 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705845 2026-02-13T02:31:54.447 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:54.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:31:54.448 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705845 2026-02-13T02:31:54.448 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:54.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705845 2026-02-13T02:31:54.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705845' 2026-02-13T02:31:54.450 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705845 2026-02-13T02:31:54.451 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:31:54.619 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705846 -lt 38654705845 2026-02-13T02:31:54.619 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:54.620 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542323 2026-02-13T02:31:54.620 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:54.621 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:31:54.622 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542323 2026-02-13T02:31:54.622 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:54.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542323 2026-02-13T02:31:54.624 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542323 2026-02-13T02:31:54.625 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542323' 2026-02-13T02:31:54.625 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:31:54.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542324 -lt 60129542323 2026-02-13T02:31:54.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:31:54.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:31:54.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:31:54.937 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:31:54.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:31:54.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:31:54.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:31:54.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:31:54.947 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:31:55.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:31:55.111 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:31:55.111 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:31:55.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:31:55.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:55.112 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:31:55.182 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836668 2026-02-13T02:31:55.182 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836668 2026-02-13T02:31:55.182 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836668' 2026-02-13T02:31:55.182 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:55.182 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:31:55.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705849 2026-02-13T02:31:55.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705849 2026-02-13T02:31:55.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836668 1-38654705849' 2026-02-13T02:31:55.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:55.257 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:31:55.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542326 2026-02-13T02:31:55.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542326 2026-02-13T02:31:55.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836668 1-38654705849 2-60129542326' 2026-02-13T02:31:55.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:55.331 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836668 2026-02-13T02:31:55.331 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:55.332 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:31:55.332 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836668 2026-02-13T02:31:55.332 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:55.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836668 2026-02-13T02:31:55.334 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836668 2026-02-13T02:31:55.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836668' 2026-02-13T02:31:55.334 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:55.498 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836665 -lt 21474836668 2026-02-13T02:31:55.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:31:56.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:31:56.500 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:56.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836668 -lt 21474836668 2026-02-13T02:31:56.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:56.665 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705849 2026-02-13T02:31:56.665 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:56.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:31:56.666 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705849 2026-02-13T02:31:56.666 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:56.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705849 2026-02-13T02:31:56.668 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705849 2026-02-13T02:31:56.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705849' 2026-02-13T02:31:56.668 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:31:56.836 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705849 -lt 38654705849 2026-02-13T02:31:56.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:56.837 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542326 2026-02-13T02:31:56.837 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:56.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:31:56.839 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542326 2026-02-13T02:31:56.839 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:56.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542326 2026-02-13T02:31:56.841 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542326 2026-02-13T02:31:56.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542326' 2026-02-13T02:31:56.841 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:31:57.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542327 -lt 60129542326 2026-02-13T02:31:57.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:31:57.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:31:57.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:31:57.158 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:31:57.167 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:31:57.167 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:31:57.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:31:57.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:31:57.168 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:31:57.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:31:57.326 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:31:57.326 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:31:57.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:31:57.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:57.326 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:31:57.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836671 2026-02-13T02:31:57.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836671 2026-02-13T02:31:57.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836671' 2026-02-13T02:31:57.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:57.399 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:31:57.474 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705852 2026-02-13T02:31:57.474 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705852 2026-02-13T02:31:57.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836671 1-38654705852' 2026-02-13T02:31:57.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:57.475 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:31:57.547 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542329 2026-02-13T02:31:57.547 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542329 2026-02-13T02:31:57.547 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836671 1-38654705852 2-60129542329' 2026-02-13T02:31:57.547 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:57.548 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836671 2026-02-13T02:31:57.548 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:57.548 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:31:57.549 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836671 2026-02-13T02:31:57.549 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:57.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836671 2026-02-13T02:31:57.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836671' 2026-02-13T02:31:57.551 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836671 2026-02-13T02:31:57.551 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:57.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836668 -lt 21474836671 2026-02-13T02:31:57.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:31:58.718 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:31:58.718 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:58.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836671 -lt 21474836671 2026-02-13T02:31:58.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:58.883 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705852 2026-02-13T02:31:58.884 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:58.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:31:58.885 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705852 2026-02-13T02:31:58.885 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:58.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705852 2026-02-13T02:31:58.887 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705852 2026-02-13T02:31:58.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705852' 2026-02-13T02:31:58.887 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:31:59.050 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705852 -lt 38654705852 2026-02-13T02:31:59.050 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:59.051 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542329 2026-02-13T02:31:59.051 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:59.051 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:31:59.052 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542329 2026-02-13T02:31:59.052 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:59.053 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542329 2026-02-13T02:31:59.054 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542329 2026-02-13T02:31:59.054 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542329' 2026-02-13T02:31:59.054 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:31:59.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542330 -lt 60129542329 2026-02-13T02:31:59.228 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:31:59.228 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:31:59.228 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:31:59.377 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:31:59.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:31:59.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:31:59.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:31:59.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:31:59.387 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:31:59.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:31:59.544 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:31:59.544 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:31:59.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:31:59.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:59.544 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:31:59.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836674 2026-02-13T02:31:59.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836674 2026-02-13T02:31:59.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836674' 2026-02-13T02:31:59.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:59.617 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:31:59.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705855 2026-02-13T02:31:59.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705855 2026-02-13T02:31:59.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836674 1-38654705855' 2026-02-13T02:31:59.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:31:59.691 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:31:59.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542333 2026-02-13T02:31:59.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542333 2026-02-13T02:31:59.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836674 1-38654705855 2-60129542333' 2026-02-13T02:31:59.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:59.762 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836674 2026-02-13T02:31:59.762 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:59.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:31:59.763 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836674 2026-02-13T02:31:59.763 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:59.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836674 2026-02-13T02:31:59.765 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836674 2026-02-13T02:31:59.766 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836674' 2026-02-13T02:31:59.766 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:31:59.925 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836674 -lt 21474836674 2026-02-13T02:31:59.925 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:31:59.926 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705855 2026-02-13T02:31:59.926 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:31:59.927 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:31:59.928 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705855 2026-02-13T02:31:59.928 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:31:59.929 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705855 2026-02-13T02:31:59.930 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705855 2026-02-13T02:31:59.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705855' 2026-02-13T02:31:59.930 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:32:00.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705855 -lt 38654705855 2026-02-13T02:32:00.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:32:00.095 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542333 2026-02-13T02:32:00.096 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:32:00.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:32:00.097 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542333 2026-02-13T02:32:00.097 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:32:00.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542333 2026-02-13T02:32:00.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542333' 2026-02-13T02:32:00.099 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542333 2026-02-13T02:32:00.099 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:32:00.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542332 -lt 60129542333 2026-02-13T02:32:00.251 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:32:01.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:32:01.252 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:32:01.414 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542332 -lt 60129542333 2026-02-13T02:32:01.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:32:02.416 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:32:02.417 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:32:02.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542335 -lt 60129542333 2026-02-13T02:32:02.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:32:02.580 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:32:02.580 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:32:02.724 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:32:02.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:32:02.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:32:02.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:32:02.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:32:02.733 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:32:02.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:32:02.889 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:32:02.889 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:32:02.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:32:02.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:32:02.890 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:32:02.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836678 2026-02-13T02:32:02.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836678 2026-02-13T02:32:02.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836678' 2026-02-13T02:32:02.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:32:02.965 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:32:03.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705859 2026-02-13T02:32:03.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705859 2026-02-13T02:32:03.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836678 1-38654705859' 2026-02-13T02:32:03.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:32:03.036 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:32:03.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542337 2026-02-13T02:32:03.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542337 2026-02-13T02:32:03.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836678 1-38654705859 2-60129542337' 2026-02-13T02:32:03.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:32:03.110 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836678 2026-02-13T02:32:03.110 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:32:03.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:32:03.111 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836678 2026-02-13T02:32:03.111 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:32:03.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836678 2026-02-13T02:32:03.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836678' 2026-02-13T02:32:03.113 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836678 2026-02-13T02:32:03.113 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:32:03.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836676 -lt 21474836678 2026-02-13T02:32:03.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:32:04.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:32:04.285 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:32:04.442 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836679 -lt 21474836678 2026-02-13T02:32:04.442 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:32:04.443 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705859 2026-02-13T02:32:04.443 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:32:04.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:32:04.444 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705859 2026-02-13T02:32:04.444 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:32:04.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705859 2026-02-13T02:32:04.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705859' 2026-02-13T02:32:04.446 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705859 2026-02-13T02:32:04.446 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:32:04.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705860 -lt 38654705859 2026-02-13T02:32:04.615 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:32:04.615 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542337 2026-02-13T02:32:04.615 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:32:04.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:32:04.617 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542337 2026-02-13T02:32:04.617 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:32:04.618 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542337 2026-02-13T02:32:04.618 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542337 2026-02-13T02:32:04.619 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542337' 2026-02-13T02:32:04.619 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:32:04.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542338 -lt 60129542337 2026-02-13T02:32:04.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:32:04.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:32:04.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:32:04.938 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:32:04.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:32:04.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:32:04.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:32:04.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:32:04.949 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:32:05.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:32:05.109 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:32:05.109 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:32:05.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:32:05.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:32:05.109 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:32:05.181 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836682 2026-02-13T02:32:05.181 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836682 2026-02-13T02:32:05.181 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836682' 2026-02-13T02:32:05.181 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:32:05.181 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:32:05.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705863 2026-02-13T02:32:05.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705863 2026-02-13T02:32:05.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836682 1-38654705863' 2026-02-13T02:32:05.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:32:05.256 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:32:05.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542340 2026-02-13T02:32:05.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542340 2026-02-13T02:32:05.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836682 1-38654705863 2-60129542340' 2026-02-13T02:32:05.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:32:05.330 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836682 2026-02-13T02:32:05.330 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:32:05.331 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:32:05.331 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836682 2026-02-13T02:32:05.332 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:32:05.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836682 2026-02-13T02:32:05.333 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836682 2026-02-13T02:32:05.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836682' 2026-02-13T02:32:05.333 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:32:05.498 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836679 -lt 21474836682 2026-02-13T02:32:05.498 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:32:06.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:32:06.500 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:32:06.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836682 -lt 21474836682 2026-02-13T02:32:06.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:32:06.666 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705863 2026-02-13T02:32:06.666 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:32:06.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:32:06.668 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705863 2026-02-13T02:32:06.668 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:32:06.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705863 2026-02-13T02:32:06.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705863' 2026-02-13T02:32:06.670 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705863 2026-02-13T02:32:06.670 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:32:06.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705863 -lt 38654705863 2026-02-13T02:32:06.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:32:06.841 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542340 2026-02-13T02:32:06.841 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:32:06.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:32:06.842 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542340 2026-02-13T02:32:06.842 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:32:06.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542340 2026-02-13T02:32:06.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542340' 2026-02-13T02:32:06.844 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542340 2026-02-13T02:32:06.845 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:32:07.006 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542341 -lt 60129542340 2026-02-13T02:32:07.006 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:32:07.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:32:07.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:32:07.152 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:32:07.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:32:07.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:32:07.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:32:07.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:32:07.162 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:32:07.315 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:32:07.315 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:32:07.316 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:32:07.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:32:07.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:32:07.316 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:32:07.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836685 2026-02-13T02:32:07.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836685 2026-02-13T02:32:07.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836685' 2026-02-13T02:32:07.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:32:07.388 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:32:07.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705866 2026-02-13T02:32:07.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705866 2026-02-13T02:32:07.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836685 1-38654705866' 2026-02-13T02:32:07.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:32:07.462 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:32:07.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542343 2026-02-13T02:32:07.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542343 2026-02-13T02:32:07.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836685 1-38654705866 2-60129542343' 2026-02-13T02:32:07.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:32:07.536 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836685 2026-02-13T02:32:07.536 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:32:07.536 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:32:07.537 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836685 2026-02-13T02:32:07.537 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:32:07.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836685 2026-02-13T02:32:07.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836685' 2026-02-13T02:32:07.539 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836685 2026-02-13T02:32:07.540 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:32:07.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836682 -lt 21474836685 2026-02-13T02:32:07.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:32:08.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:32:08.699 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:32:08.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836685 -lt 21474836685 2026-02-13T02:32:08.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:32:08.866 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705866 2026-02-13T02:32:08.866 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:32:08.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:32:08.868 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705866 2026-02-13T02:32:08.868 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:32:08.869 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705866 2026-02-13T02:32:08.870 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705866 2026-02-13T02:32:08.870 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705866' 2026-02-13T02:32:08.870 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:32:09.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705866 -lt 38654705866 2026-02-13T02:32:09.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:32:09.041 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542343 2026-02-13T02:32:09.041 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:32:09.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:32:09.043 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542343 2026-02-13T02:32:09.044 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:32:09.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542343 2026-02-13T02:32:09.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542343' 2026-02-13T02:32:09.045 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542343 2026-02-13T02:32:09.045 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:32:09.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542344 -lt 60129542343 2026-02-13T02:32:09.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:32:09.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:32:09.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:32:09.363 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:32:09.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:32:09.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:32:09.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:32:09.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:32:09.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:32:09.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:32:09.538 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:32:09.538 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:32:09.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:32:09.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:32:09.539 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:32:09.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836688 2026-02-13T02:32:09.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836688 2026-02-13T02:32:09.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836688' 2026-02-13T02:32:09.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:32:09.609 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:32:09.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705869 2026-02-13T02:32:09.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705869 2026-02-13T02:32:09.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836688 1-38654705869' 2026-02-13T02:32:09.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:32:09.685 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:32:09.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542347 2026-02-13T02:32:09.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542347 2026-02-13T02:32:09.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836688 1-38654705869 2-60129542347' 2026-02-13T02:32:09.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:32:09.757 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836688 2026-02-13T02:32:09.757 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:32:09.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:32:09.759 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836688 2026-02-13T02:32:09.759 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:32:09.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836688 2026-02-13T02:32:09.761 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836688 2026-02-13T02:32:09.762 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836688' 2026-02-13T02:32:09.762 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:32:09.928 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836688 -lt 21474836688 2026-02-13T02:32:09.928 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:32:09.929 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705869 2026-02-13T02:32:09.929 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:32:09.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:32:09.931 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705869 2026-02-13T02:32:09.931 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:32:09.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705869 2026-02-13T02:32:09.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705869' 2026-02-13T02:32:09.933 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705869 2026-02-13T02:32:09.933 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:32:10.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705869 -lt 38654705869 2026-02-13T02:32:10.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:32:10.102 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542347 2026-02-13T02:32:10.102 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:32:10.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:32:10.104 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542347 2026-02-13T02:32:10.104 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:32:10.105 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542347 2026-02-13T02:32:10.106 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542347 2026-02-13T02:32:10.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542347' 2026-02-13T02:32:10.106 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:32:10.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542346 -lt 60129542347 2026-02-13T02:32:10.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:32:11.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:32:11.276 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:32:11.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542346 -lt 60129542347 2026-02-13T02:32:11.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:32:12.435 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:32:12.435 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:32:12.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542349 -lt 60129542347 2026-02-13T02:32:12.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:32:12.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:32:12.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:32:12.749 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:32:12.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: break 2026-02-13T02:32:12.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: set +o pipefail 2026-02-13T02:32:12.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: sleep 5 2026-02-13T02:32:17.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:384: _scrub_abort: grep 'nodeep_scrub set, aborting' td/osd-scrub-test/osd.1.log 2026-02-13T02:32:17.768 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T02:30:15.726+0000 7f107809d640 10 osd.1 pg_epoch: 20 pg[1.0( v 18'1000 (0'0,18'1000] local-lis/les=15/17 n=1000 ec=15/15 lis/c=15/15 les/c/f=17/17/0 sis=15) [1,0,2] r=0 lpr=15 crt=18'1000 lcod 18'999 mlcod 18'999 active+clean+scrubbing+deep(periodic-regular)] scrubber: nodeep_scrub set, aborting 2026-02-13T02:32:17.769 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:390: _scrub_abort: get_last_scrub_stamp 1.0 2026-02-13T02:32:17.770 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:32:17.770 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:32:17.770 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:32:17.770 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:32:17.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:390: _scrub_abort: local last_scrub=2026-02-13T02:29:42.203241+0000 2026-02-13T02:32:17.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:391: _scrub_abort: ceph config set osd osd_scrub_sleep 0.1 2026-02-13T02:32:18.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:393: _scrub_abort: ceph osd unset nodeep-scrub 2026-02-13T02:32:18.283 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is unset 2026-02-13T02:32:18.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:394: _scrub_abort: '[' deep-scrub = deep-scrub ']' 2026-02-13T02:32:18.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:396: _scrub_abort: ceph osd unset noscrub 2026-02-13T02:32:18.484 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is unset 2026-02-13T02:32:18.498 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:398: _scrub_abort: TIMEOUT=500 2026-02-13T02:32:18.498 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:399: _scrub_abort: wait_for_scrub 1.0 2026-02-13T02:29:42.203241+0000 2026-02-13T02:32:18.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-13T02:32:18.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:29:42.203241+0000 2026-02-13T02:32:18.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:32:18.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:32:18.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:32:18.499 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:32:18.499 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:32:18.499 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:32:18.499 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:32:18.499 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:32:18.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:32:18.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:32:19.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:32:19.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:32:19.661 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:32:19.661 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:32:19.662 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:32:19.662 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:32:19.662 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:32:19.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:32:19.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:32:20.827 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:32:20.827 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:32:20.828 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:32:20.828 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:32:20.828 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:32:20.828 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:32:20.828 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:32:20.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:32:20.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:32:21.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:32:21.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:32:21.994 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:32:21.994 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:32:21.995 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:32:21.995 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:32:21.995 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:32:22.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:32:22.152 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:32:23.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:32:23.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:32:23.154 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:32:23.154 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:32:23.154 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:32:23.154 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:32:23.154 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:32:23.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:32:23.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:32:24.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:32:24.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:32:24.309 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:32:24.309 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:32:24.309 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:32:24.310 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:32:24.310 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:32:24.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:32:24.471 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:32:25.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:32:25.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:32:25.473 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:32:25.473 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:32:25.473 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:32:25.473 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:32:25.474 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:32:25.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:32:25.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:32:26.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:32:26.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:32:26.633 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:32:26.633 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:32:26.633 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:32:26.633 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:32:26.633 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:32:26.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:32:26.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:32:27.801 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:32:27.801 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:32:27.801 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:32:27.801 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:32:27.801 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:32:27.802 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:32:27.802 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:32:27.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:32:27.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:32:28.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:32:28.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:32:28.964 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:32:28.965 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:32:28.965 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:32:28.965 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:32:28.965 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:32:29.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:32:29.128 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:32:30.128 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:32:30.129 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:32:30.130 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:32:30.130 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:32:30.130 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:32:30.130 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:32:30.130 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:32:30.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:32:30.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:32:31.289 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:32:31.289 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:32:31.289 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:32:31.289 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:32:31.290 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:32:31.290 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:32:31.290 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:32:31.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:32:31.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:32:32.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:32:32.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:32:32.454 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:32:32.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:32:32.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:32:32.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:32:32.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:32:32.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:32:32.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:32:33.615 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:32:33.615 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:32:33.616 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:32:33.616 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:32:33.617 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:32:33.617 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:32:33.617 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:32:33.773 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:32:33.773 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:32:34.774 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:32:34.775 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:32:34.775 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:32:34.775 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:32:34.775 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:32:34.775 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:32:34.776 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:32:34.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:32:34.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:32:35.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:32:35.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:32:35.932 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:32:35.932 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:32:35.932 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:32:35.933 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:32:35.933 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:32:36.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:32:36.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:32:37.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:32:37.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:32:37.089 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:32:37.089 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:32:37.090 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:32:37.090 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:32:37.090 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:32:37.255 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:32:37.255 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:32:38.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:32:38.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:32:38.258 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:32:38.258 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:32:38.258 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:32:38.258 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:32:38.258 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:32:38.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:32:38.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:32:39.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:32:39.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:32:39.421 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:32:39.421 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:32:39.421 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:32:39.422 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:32:39.422 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:32:39.581 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:32:39.581 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:32:40.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:32:40.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:32:40.583 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:32:40.584 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:32:40.584 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:32:40.584 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:32:40.584 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:32:40.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:32:40.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:32:41.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:32:41.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:32:41.743 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:32:41.743 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:32:41.743 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:32:41.744 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:32:41.744 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:32:41.899 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:32:41.899 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:32:42.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:32:42.901 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:32:42.902 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:32:42.902 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:32:42.902 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:32:42.902 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:32:42.902 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:32:43.064 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:32:43.064 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:32:44.066 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:32:44.066 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:32:44.067 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:32:44.067 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:32:44.067 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:32:44.067 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:32:44.067 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:32:44.230 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:32:44.230 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:32:45.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:32:45.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:32:45.232 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:32:45.232 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:32:45.232 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:32:45.232 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:32:45.232 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:32:45.393 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:32:45.393 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:32:46.394 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:32:46.394 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:32:46.395 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:32:46.395 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:32:46.395 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:32:46.395 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:32:46.395 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:32:46.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:32:46.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:32:47.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:32:47.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:32:47.553 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:32:47.553 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:32:47.553 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:32:47.553 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:32:47.553 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:32:47.707 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:32:47.707 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:32:48.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:32:48.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:32:48.709 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:32:48.709 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:32:48.709 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:32:48.710 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:32:48.710 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:32:48.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:32:48.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:32:49.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:32:49.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:32:49.869 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:32:49.869 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:32:49.870 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:32:49.870 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:32:49.870 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:32:50.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:32:50.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:32:51.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:32:51.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:32:51.029 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:32:51.029 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:32:51.029 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:32:51.029 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:32:51.029 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:32:51.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:32:51.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:32:52.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:32:52.190 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:32:52.190 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:32:52.191 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:32:52.191 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:32:52.191 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:32:52.191 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:32:52.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:32:52.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:32:53.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:32:53.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:32:53.353 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:32:53.353 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:32:53.354 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:32:53.354 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:32:53.354 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:32:53.516 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:32:53.516 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:32:54.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:32:54.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:32:54.519 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:32:54.519 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:32:54.519 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:32:54.519 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:32:54.519 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:32:54.678 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:32:54.678 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:32:55.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:32:55.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:32:55.680 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:32:55.680 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:32:55.680 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:32:55.680 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:32:55.681 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:32:55.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:32:55.845 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:32:56.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:32:56.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:32:56.847 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:32:56.847 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:32:56.847 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:32:56.847 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:32:56.847 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:32:57.006 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:32:57.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:32:58.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:32:58.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:32:58.009 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:32:58.009 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:32:58.009 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:32:58.009 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:32:58.009 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:32:58.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:32:58.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:32:59.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:32:59.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:32:59.170 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:32:59.171 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:32:59.171 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:32:59.171 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:32:59.171 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:32:59.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:32:59.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:00.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:00.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:00.329 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:00.329 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:00.329 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:00.329 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:00.329 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:00.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:00.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:01.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:01.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:01.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:01.489 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:01.489 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:01.489 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:01.489 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:01.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:01.642 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:02.643 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:02.643 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:02.643 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:02.644 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:02.644 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:02.644 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:02.644 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:02.797 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:02.797 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:03.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:03.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:03.799 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:03.800 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:03.800 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:03.800 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:03.800 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:03.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:03.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:04.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:04.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:04.963 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:04.964 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:04.964 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:04.964 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:04.964 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:05.125 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:05.125 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:06.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:06.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:06.127 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:06.127 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:06.127 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:06.127 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:06.127 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:06.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:06.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:07.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:07.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:07.287 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:07.288 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:07.288 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:07.288 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:07.288 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:07.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:07.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:08.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:08.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:08.452 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:08.452 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:08.452 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:08.452 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:08.452 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:08.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:08.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:09.615 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:09.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:09.616 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:09.617 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:09.617 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:09.617 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:09.617 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:09.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:09.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:10.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:10.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:10.779 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:10.780 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:10.780 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:10.780 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:10.780 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:10.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:10.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:11.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:11.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:11.941 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:11.942 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:11.942 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:11.942 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:11.942 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:12.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:12.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:13.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:13.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:13.102 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:13.102 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:13.103 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:13.103 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:13.103 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:13.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:13.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:14.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:14.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:14.263 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:14.263 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:14.263 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:14.264 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:14.264 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:14.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:14.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:15.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:15.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:15.429 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:15.429 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:15.429 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:15.430 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:15.430 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:15.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:15.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:16.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:16.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:16.593 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:16.593 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:16.593 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:16.593 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:16.593 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:16.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:16.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:17.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:17.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:17.755 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:17.756 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:17.756 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:17.756 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:17.756 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:17.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:17.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:18.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:18.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:18.919 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:18.919 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:18.919 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:18.920 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:18.920 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:19.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:19.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:20.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:20.079 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:20.080 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:20.080 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:20.080 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:20.080 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:20.080 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:20.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:20.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:21.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:21.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:21.238 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:21.238 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:21.238 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:21.239 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:21.239 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:21.395 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:21.395 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:22.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:22.397 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:22.397 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:22.397 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:22.397 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:22.398 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:22.398 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:22.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:22.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:23.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:23.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:23.563 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:23.563 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:23.563 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:23.563 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:23.563 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:23.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:23.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:24.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:24.723 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:24.724 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:24.724 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:24.724 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:24.724 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:24.724 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:24.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:24.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:25.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:25.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:25.885 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:25.886 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:25.886 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:25.886 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:25.886 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:26.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:26.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:27.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:27.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:27.045 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:27.045 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:27.045 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:27.045 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:27.046 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:27.202 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:27.202 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:28.203 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:28.203 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:28.204 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:28.204 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:28.204 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:28.204 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:28.204 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:28.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:28.372 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:29.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:29.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:29.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:29.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:29.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:29.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:29.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:29.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:29.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:30.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:30.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:30.541 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:30.541 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:30.541 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:30.541 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:30.541 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:30.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:30.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:31.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:31.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:31.701 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:31.701 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:31.701 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:31.701 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:31.701 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:31.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:31.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:32.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:32.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:32.863 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:32.864 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:32.864 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:32.864 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:32.864 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:33.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:33.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:34.024 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:34.024 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:34.025 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:34.025 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:34.025 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:34.025 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:34.025 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:34.184 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:34.184 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:35.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:35.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:35.187 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:35.187 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:35.187 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:35.187 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:35.187 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:35.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:35.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:36.343 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:36.343 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:36.343 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:36.343 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:36.343 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:36.344 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:36.344 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:36.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:36.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:37.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:37.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:37.502 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:37.502 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:37.502 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:37.502 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:37.502 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:37.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:37.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:38.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:38.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:38.663 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:38.663 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:38.663 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:38.663 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:38.663 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:38.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:38.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:39.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:39.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:39.823 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:39.823 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:39.823 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:39.823 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:39.823 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:39.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:39.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:40.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:40.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:40.979 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:40.979 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:40.979 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:40.979 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:40.979 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:41.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:41.132 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:42.132 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:42.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:42.134 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:42.134 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:42.134 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:42.134 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:42.134 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:42.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:42.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:43.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:43.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:43.288 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:43.289 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:43.289 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:43.289 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:43.289 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:43.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:43.441 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:44.442 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:44.442 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:44.443 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:44.443 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:44.443 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:44.443 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:44.443 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:44.597 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:44.597 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:45.598 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:45.598 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:45.599 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:45.600 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:45.600 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:45.600 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:45.600 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:45.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:45.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:46.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:46.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:46.755 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:46.755 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:46.755 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:46.756 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:46.756 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:46.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:46.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:47.910 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:47.910 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:47.911 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:47.911 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:47.911 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:47.912 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:47.912 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:48.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:48.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:49.064 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:49.064 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:49.065 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:49.065 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:49.065 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:49.065 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:49.065 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:49.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:49.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:50.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:50.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:50.228 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:50.228 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:50.229 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:50.229 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:50.229 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:50.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:50.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:51.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:51.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:51.391 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:51.391 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:51.391 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:51.391 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:51.391 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:51.547 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:51.547 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:52.548 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:52.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:52.549 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:52.549 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:52.549 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:52.549 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:52.550 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:52.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:52.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:53.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:53.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:53.712 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:53.712 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:53.713 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:53.713 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:53.713 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:53.872 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:53.872 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:54.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:54.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:54.874 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:54.874 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:54.874 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:54.875 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:54.875 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:55.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:55.031 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:56.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:56.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:56.033 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:56.033 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:56.034 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:56.034 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:56.034 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:56.190 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:56.190 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:57.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:57.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:57.192 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:57.192 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:57.192 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:57.192 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:57.193 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:57.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:57.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:58.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:58.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:58.351 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:58.351 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:58.351 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:58.351 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:58.351 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:58.513 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:58.513 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:33:59.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:33:59.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:33:59.515 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:33:59.515 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:33:59.515 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:33:59.516 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:33:59.516 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:33:59.680 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:33:59.681 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:00.681 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:00.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:00.682 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:00.682 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:00.683 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:00.683 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:00.683 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:00.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:00.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:01.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:01.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:01.843 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:01.843 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:01.843 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:01.843 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:01.843 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:02.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:02.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:03.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:03.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:03.004 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:03.004 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:03.004 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:03.005 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:03.005 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:03.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:03.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:04.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:04.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:04.166 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:04.166 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:04.166 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:04.166 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:04.166 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:04.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:04.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:05.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:05.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:05.326 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:05.327 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:05.327 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:05.327 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:05.327 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:05.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:05.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:06.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:06.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:06.482 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:06.482 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:06.482 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:06.483 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:06.483 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:06.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:06.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:07.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:07.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:07.637 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:07.637 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:07.637 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:07.637 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:07.637 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:07.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:07.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:08.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:08.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:08.794 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:08.795 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:08.795 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:08.795 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:08.795 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:08.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:08.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:09.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:09.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:09.953 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:09.953 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:09.953 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:09.954 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:09.954 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:10.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:10.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:11.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:11.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:11.112 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:11.112 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:11.113 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:11.113 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:11.113 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:11.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:11.266 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:12.267 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:12.267 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:12.268 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:12.268 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:12.268 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:12.268 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:12.268 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:12.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:12.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:13.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:13.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:13.426 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:13.427 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:13.427 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:13.427 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:13.427 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:13.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:13.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:14.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:14.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:14.590 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:14.591 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:14.591 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:14.591 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:14.591 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:14.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:14.747 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:15.748 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:15.748 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:15.749 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:15.749 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:15.749 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:15.749 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:15.749 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:15.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:15.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:16.910 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:16.910 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:16.911 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:16.911 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:16.911 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:16.911 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:16.911 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:17.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:17.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:18.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:18.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:18.070 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:18.070 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:18.070 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:18.070 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:18.071 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:18.229 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:18.229 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:19.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:19.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:19.232 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:19.232 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:19.232 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:19.232 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:19.233 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:19.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:19.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:20.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:20.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:20.392 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:20.392 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:20.392 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:20.393 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:20.393 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:20.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:20.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:21.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:21.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:21.546 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:21.546 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:21.546 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:21.546 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:21.546 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:21.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:21.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:22.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:22.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:22.704 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:22.704 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:22.704 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:22.704 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:22.704 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:22.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:22.857 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:23.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:23.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:23.859 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:23.859 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:23.859 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:23.859 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:23.859 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:24.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:24.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:25.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:25.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:25.021 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:25.021 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:25.022 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:25.022 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:25.022 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:25.184 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:25.184 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:26.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:26.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:26.187 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:26.187 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:26.187 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:26.187 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:26.187 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:26.348 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:26.348 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:27.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:27.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:27.351 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:27.351 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:27.351 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:27.351 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:27.351 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:27.511 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:27.511 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:28.512 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:28.512 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:28.513 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:28.513 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:28.514 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:28.514 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:28.514 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:28.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:28.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:29.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:29.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:29.676 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:29.677 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:29.677 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:29.677 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:29.677 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:29.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:29.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:30.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:30.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:30.832 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:30.832 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:30.833 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:30.833 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:30.833 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:30.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:30.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:31.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:31.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:31.998 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:31.998 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:31.998 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:31.998 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:31.999 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:32.155 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:32.155 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:33.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:33.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:33.158 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:33.158 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:33.158 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:33.158 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:33.158 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:33.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:33.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:34.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:34.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:34.314 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:34.314 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:34.314 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:34.314 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:34.314 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:34.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:34.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:35.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:35.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:35.477 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:35.477 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:35.477 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:35.477 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:35.477 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:35.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:35.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:36.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:36.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:36.638 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:36.638 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:36.638 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:36.639 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:36.639 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:36.797 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:36.797 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:37.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:37.799 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:37.799 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:37.799 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:37.799 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:37.800 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:37.800 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:37.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:37.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:38.959 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:38.959 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:38.959 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:38.959 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:38.959 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:38.960 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:38.960 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:39.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:39.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:40.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:40.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:40.116 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:40.116 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:40.116 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:40.117 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:40.117 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:40.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:40.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:41.280 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:41.280 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:41.280 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:41.280 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:41.280 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:41.280 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:41.281 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:41.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:41.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:42.441 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:42.441 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:42.441 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:42.441 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:42.441 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:42.442 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:42.442 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:42.598 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:42.598 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:43.599 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:43.599 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:43.600 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:43.600 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:43.600 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:43.600 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:43.601 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:43.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:43.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:44.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:44.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:44.759 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:44.759 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:44.759 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:44.760 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:44.760 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:44.915 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:44.915 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:45.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:45.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:45.916 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:45.917 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:45.917 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:45.917 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:45.917 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:46.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:46.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:47.082 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:47.082 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:47.083 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:47.083 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:47.083 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:47.083 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:47.083 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:47.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:47.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:48.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:48.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:48.247 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:48.247 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:48.247 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:48.247 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:48.247 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:48.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:48.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:49.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:49.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:49.406 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:49.406 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:49.406 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:49.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:49.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:49.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:49.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:50.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:50.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:50.567 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:50.567 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:50.567 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:50.567 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:50.567 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:50.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:50.723 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:51.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:51.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:51.725 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:51.725 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:51.725 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:51.726 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:51.726 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:51.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:51.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:52.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:52.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:52.883 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:52.884 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:52.884 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:52.884 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:52.885 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:53.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:53.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:54.042 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:54.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:54.044 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:54.044 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:54.044 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:54.044 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:54.044 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:54.198 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:54.198 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:55.199 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:55.199 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:55.200 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:55.200 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:55.200 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:55.201 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:55.201 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:55.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:55.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:56.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:56.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:56.364 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:56.365 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:56.365 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:56.365 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:56.365 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:56.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:56.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:57.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:57.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:57.527 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:57.527 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:57.527 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:57.527 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:57.527 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:57.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:57.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:58.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:58.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:58.684 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:58.684 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:58.684 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:58.685 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:58.685 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:34:58.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:34:58.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:34:59.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:34:59.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:34:59.844 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:34:59.844 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:34:59.845 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:34:59.845 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:34:59.845 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:00.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:00.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:01.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:01.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:01.003 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:01.003 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:01.003 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:01.003 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:01.003 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:01.154 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:01.155 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:02.155 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:02.155 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:02.156 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:02.156 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:02.156 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:02.157 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:02.157 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:02.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:02.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:03.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:03.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:03.322 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:03.322 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:03.322 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:03.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:03.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:03.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:03.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:04.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:04.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:04.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:04.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:04.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:04.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:04.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:04.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:04.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:05.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:05.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:05.650 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:05.650 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:05.650 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:05.650 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:05.650 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:05.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:05.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:06.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:06.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:06.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:06.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:06.812 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:06.812 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:06.812 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:06.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:06.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:07.970 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:07.971 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:07.972 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:07.972 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:07.972 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:07.972 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:07.972 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:08.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:08.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:09.132 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:09.132 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:09.133 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:09.133 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:09.133 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:09.133 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:09.133 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:09.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:09.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:10.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:10.289 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:10.290 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:10.290 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:10.290 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:10.290 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:10.290 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:10.442 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:10.442 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:11.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:11.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:11.444 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:11.444 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:11.445 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:11.445 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:11.445 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:11.597 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:11.598 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:12.599 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:12.599 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:12.599 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:12.599 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:12.600 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:12.600 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:12.601 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:12.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:12.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:13.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:13.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:13.756 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:13.756 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:13.756 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:13.757 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:13.757 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:13.910 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:13.910 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:14.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:14.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:14.912 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:14.912 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:14.912 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:14.912 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:14.912 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:15.065 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:15.066 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:16.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:16.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:16.068 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:16.068 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:16.068 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:16.068 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:16.068 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:16.223 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:16.224 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:17.224 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:17.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:17.226 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:17.226 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:17.226 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:17.226 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:17.226 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:17.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:17.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:18.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:18.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:18.389 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:18.389 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:18.389 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:18.389 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:18.389 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:18.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:18.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:19.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:19.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:19.552 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:19.552 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:19.552 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:19.552 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:19.552 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:19.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:19.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:20.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:20.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:20.713 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:20.713 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:20.713 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:20.713 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:20.713 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:20.871 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:20.871 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:21.872 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:21.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:21.873 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:21.873 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:21.873 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:21.873 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:21.873 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:22.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:22.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:23.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:23.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:23.029 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:23.029 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:23.029 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:23.030 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:23.030 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:23.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:23.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:24.190 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:24.190 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:24.191 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:24.191 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:24.191 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:24.191 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:24.192 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:24.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:24.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:25.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:25.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:25.354 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:25.354 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:25.354 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:25.355 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:25.355 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:25.515 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:25.515 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:26.516 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:26.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:26.517 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:26.518 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:26.518 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:26.518 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:26.518 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:26.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:26.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:27.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:27.678 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:27.678 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:27.678 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:27.679 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:27.679 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:27.679 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:27.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:27.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:28.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:28.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:28.834 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:28.834 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:28.834 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:28.834 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:28.834 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:28.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:28.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:29.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:29.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:29.997 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:29.997 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:29.997 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:29.998 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:29.998 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:30.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:30.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:31.160 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:31.160 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:31.161 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:31.161 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:31.161 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:31.161 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:31.161 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:31.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:31.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:32.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:32.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:32.326 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:32.326 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:32.326 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:32.326 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:32.326 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:32.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:32.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:33.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:33.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:33.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:33.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:33.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:33.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:33.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:33.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:33.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:34.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:34.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:34.651 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:34.651 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:34.651 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:34.651 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:34.651 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:34.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:34.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:35.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:35.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:35.813 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:35.813 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:35.813 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:35.813 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:35.813 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:35.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:35.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:36.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:36.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:36.974 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:36.975 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:36.975 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:36.975 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:36.975 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:37.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:37.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:38.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:38.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:38.132 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:38.132 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:38.132 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:38.133 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:38.133 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:38.291 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:38.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:39.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:39.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:39.293 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:39.294 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:39.294 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:39.294 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:39.294 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:39.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:39.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:40.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:40.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:40.452 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:40.452 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:40.452 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:40.452 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:40.452 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:40.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:40.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:41.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:41.610 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:41.610 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:41.611 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:41.611 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:41.611 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:41.611 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:41.768 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:41.769 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:42.770 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:42.770 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:42.771 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:42.771 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:42.771 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:42.771 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:42.772 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:42.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:42.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:43.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:43.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:43.935 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:43.935 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:43.935 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:43.936 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:43.936 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:44.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:44.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:45.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:45.099 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:45.100 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:45.100 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:45.100 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:45.100 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:45.100 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:45.258 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:45.258 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:46.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:46.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:46.260 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:46.260 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:46.260 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:46.260 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:46.260 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:46.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:46.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:47.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:47.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:47.423 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:47.423 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:47.424 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:47.424 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:47.424 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:47.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:47.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:48.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:48.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:48.580 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:48.580 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:48.580 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:48.580 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:48.580 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:48.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:48.740 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:49.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:49.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:49.742 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:49.742 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:49.742 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:49.742 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:49.743 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:49.898 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:49.898 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:50.899 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:50.899 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:50.900 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:50.901 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:50.901 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:50.901 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:50.901 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:51.058 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:51.058 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:52.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:52.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:52.060 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:52.060 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:52.060 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:52.060 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:52.060 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:52.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:52.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:53.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:53.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:53.217 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:53.218 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:53.218 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:53.218 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:53.218 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:53.372 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:53.372 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:54.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:54.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:54.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:54.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:54.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:54.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:54.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:54.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:54.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:55.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:55.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:55.532 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:55.532 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:55.532 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:55.532 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:55.532 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:55.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:55.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:56.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:56.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:56.687 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:56.688 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:56.688 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:56.688 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:56.688 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:56.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:56.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:57.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:57.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:57.845 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:57.845 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:57.845 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:57.845 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:57.845 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:58.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:58.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:35:59.006 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:35:59.006 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:35:59.007 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:35:59.007 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:35:59.007 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:35:59.007 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:35:59.008 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:35:59.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:35:59.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:00.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:00.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:00.167 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:00.167 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:00.167 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:00.167 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:00.167 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:00.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:00.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:01.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:01.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:01.326 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:01.326 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:01.326 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:01.326 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:01.327 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:01.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:01.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:02.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:02.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:02.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:02.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:02.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:02.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:02.489 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:02.646 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:02.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:03.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:03.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:03.648 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:03.649 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:03.649 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:03.649 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:03.649 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:03.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:03.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:04.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:04.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:04.810 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:04.810 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:04.810 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:04.810 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:04.810 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:04.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:04.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:05.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:05.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:05.970 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:05.970 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:05.970 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:05.970 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:05.970 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:06.129 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:06.129 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:07.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:07.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:07.131 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:07.131 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:07.131 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:07.131 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:07.131 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:07.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:07.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:08.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:08.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:08.289 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:08.289 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:08.289 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:08.289 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:08.289 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:08.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:08.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:09.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:09.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:09.449 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:09.449 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:09.449 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:09.450 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:09.450 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:09.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:09.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:10.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:10.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:10.604 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:10.604 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:10.604 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:10.605 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:10.605 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:10.762 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:10.762 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:11.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:11.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:11.764 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:11.764 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:11.764 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:11.764 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:11.764 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:11.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:11.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:12.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:12.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:12.922 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:12.922 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:12.923 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:12.923 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:12.923 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:13.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:13.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:14.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:14.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:14.078 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:14.079 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:14.079 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:14.079 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:14.079 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:14.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:14.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:15.232 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:15.233 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:15.233 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:15.234 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:15.234 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:15.234 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:15.234 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:15.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:15.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:16.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:16.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:16.391 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:16.391 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:16.391 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:16.391 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:16.391 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:16.547 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:16.547 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:17.548 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:17.548 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:17.549 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:17.549 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:17.549 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:17.549 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:17.550 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:17.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:17.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:18.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:18.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:18.703 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:18.704 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:18.704 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:18.704 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:18.704 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:18.857 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:18.857 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:19.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:19.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:19.859 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:19.859 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:19.860 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:19.860 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:19.860 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:20.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:20.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:21.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:21.015 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:21.016 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:21.016 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:21.016 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:21.016 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:21.016 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:21.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:21.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:22.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:22.170 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:22.170 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:22.171 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:22.171 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:22.171 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:22.171 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:22.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:22.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:23.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:23.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:23.327 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:23.328 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:23.328 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:23.328 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:23.328 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:23.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:23.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:24.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:24.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:24.484 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:24.484 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:24.484 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:24.485 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:24.485 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:24.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:24.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:25.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:25.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:25.642 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:25.642 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:25.642 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:25.643 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:25.643 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:25.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:25.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:26.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:26.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:26.806 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:26.806 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:26.806 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:26.806 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:26.806 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:26.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:26.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:27.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:27.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:27.967 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:27.968 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:27.968 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:27.968 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:27.968 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:28.123 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:28.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:29.125 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:29.125 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:29.126 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:29.126 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:29.126 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:29.126 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:29.126 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:29.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:29.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:30.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:30.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:30.288 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:30.288 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:30.288 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:30.288 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:30.288 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:30.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:30.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:31.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:31.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:31.449 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:31.449 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:31.450 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:31.450 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:31.450 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:31.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:31.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:32.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:32.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:32.608 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:32.608 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:32.608 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:32.608 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:32.608 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:32.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:32.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:33.774 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:33.774 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:33.775 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:33.775 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:33.775 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:33.775 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:33.775 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:33.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:33.932 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:34.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:34.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:34.934 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:34.934 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:34.934 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:34.934 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:34.935 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:35.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:35.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:36.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:36.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:36.093 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:36.093 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:36.093 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:36.094 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:36.094 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:36.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:36.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:37.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:37.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:37.253 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:37.253 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:37.253 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:37.253 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:37.254 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:37.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:37.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:38.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:38.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:38.409 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:38.409 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:38.409 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:38.410 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:38.410 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:38.572 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:38.572 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:39.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:39.574 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:39.575 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:39.575 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:39.575 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:39.575 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:39.575 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:39.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:39.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:40.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:40.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:40.734 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:40.734 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:40.734 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:40.735 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:40.735 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:40.893 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:40.893 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:41.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:41.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:41.895 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:41.895 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:41.895 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:41.896 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:41.896 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:42.057 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:42.057 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:43.058 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:43.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:43.059 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:43.060 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:43.060 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:43.060 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:43.060 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:43.219 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:43.219 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:44.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:44.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:44.221 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:44.221 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:44.221 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:44.221 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:44.221 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:44.383 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:44.383 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:45.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:45.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:45.385 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:45.386 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:45.386 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:45.386 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:45.386 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:45.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:45.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:46.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:46.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:46.554 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:46.554 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:46.554 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:46.554 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:46.554 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:46.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:46.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:47.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:47.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:47.715 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:47.715 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:47.715 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:47.716 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:47.716 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:47.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:47.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:48.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:48.876 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:48.877 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:48.877 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:48.877 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:48.877 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:48.877 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:49.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:49.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:50.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:50.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:50.040 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:50.040 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:50.041 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:50.041 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:50.041 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:50.201 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:50.201 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:51.202 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:51.202 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:51.203 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:51.203 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:51.203 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:51.203 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:51.204 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:51.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:51.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:52.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:52.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:52.365 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:52.365 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:52.365 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:52.365 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:52.365 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:52.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:52.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:53.527 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:53.527 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:53.528 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:53.528 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:53.528 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:53.528 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:53.529 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:53.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:53.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:54.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:54.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:54.691 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:54.692 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:54.692 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:54.692 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:54.692 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:54.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:54.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:55.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:55.852 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:55.853 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:55.853 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:55.853 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:55.853 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:55.853 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:56.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:56.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:57.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:57.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:57.014 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:57.014 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:57.014 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:57.014 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:57.014 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:57.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:57.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:58.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:58.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:58.174 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:58.174 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:58.174 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:58.174 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:58.174 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:58.332 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:58.332 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:36:59.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:36:59.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:36:59.334 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:36:59.334 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:36:59.334 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:36:59.335 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:36:59.335 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:36:59.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:36:59.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:00.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:00.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:00.494 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:00.494 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:00.494 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:00.494 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:00.495 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:00.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:00.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:01.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:01.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:01.653 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:01.653 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:01.653 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:01.653 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:01.653 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:01.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:01.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:02.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:02.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:02.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:02.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:02.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:02.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:02.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:02.971 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:02.971 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:03.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:03.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:03.973 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:03.974 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:03.974 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:03.974 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:03.974 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:04.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:04.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:05.136 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:05.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:05.138 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:05.138 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:05.138 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:05.138 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:05.138 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:05.295 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:05.295 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:06.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:06.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:06.297 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:06.298 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:06.298 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:06.298 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:06.298 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:06.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:06.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:07.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:07.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:07.460 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:07.460 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:07.460 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:07.460 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:07.460 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:07.620 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:07.621 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:08.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:08.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:08.623 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:08.623 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:08.623 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:08.623 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:08.623 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:08.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:08.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:09.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:09.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:09.778 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:09.778 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:09.778 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:09.778 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:09.778 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:09.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:09.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:10.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:10.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:10.936 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:10.936 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:10.936 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:10.936 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:10.936 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:11.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:11.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:12.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:12.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:12.093 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:12.093 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:12.093 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:12.094 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:12.094 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:12.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:12.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:13.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:13.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:13.246 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:13.247 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:13.247 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:13.247 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:13.247 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:13.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:13.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:14.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:14.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:14.401 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:14.402 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:14.402 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:14.402 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:14.402 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:14.558 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:14.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:15.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:15.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:15.561 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:15.561 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:15.561 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:15.561 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:15.561 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:15.720 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:15.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:16.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:16.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:16.723 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:16.723 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:16.724 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:16.724 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:16.724 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:16.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:16.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:17.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:17.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:17.882 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:17.883 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:17.883 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:17.883 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:17.883 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:18.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:18.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:19.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:19.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:19.042 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:19.043 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:19.043 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:19.043 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:19.043 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:19.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:19.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:20.201 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:20.201 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:20.202 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:20.202 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:20.202 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:20.203 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:20.203 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:20.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:20.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:21.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:21.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:21.363 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:21.363 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:21.363 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:21.363 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:21.363 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:21.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:21.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:22.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:22.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:22.521 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:22.521 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:22.521 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:22.521 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:22.522 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:22.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:22.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:23.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:23.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:23.677 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:23.677 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:23.677 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:23.677 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:23.678 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:23.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:23.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:24.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:24.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:24.835 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:24.835 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:24.835 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:24.835 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:24.835 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:24.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:24.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:25.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:25.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:25.993 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:25.993 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:25.993 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:25.993 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:25.994 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:26.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:26.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:27.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:27.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:27.152 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:27.152 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:27.153 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:27.153 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:27.153 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:27.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:27.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:28.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:28.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:28.311 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:28.312 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:28.312 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:28.312 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:28.312 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:28.465 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:28.465 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:29.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:29.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:29.467 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:29.467 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:29.467 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:29.467 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:29.468 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:29.620 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:29.620 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:30.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:30.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:30.623 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:30.623 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:30.623 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:30.623 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:30.623 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:30.779 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:30.779 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:31.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:31.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:31.781 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:31.781 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:31.781 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:31.782 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:31.782 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:31.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:31.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:32.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:32.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:32.937 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:32.938 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:32.938 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:32.938 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:32.938 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:33.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:33.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:34.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:34.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:34.103 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:34.103 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:34.103 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:34.103 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:34.103 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:34.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:34.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:35.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:35.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:35.266 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:35.266 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:35.267 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:35.267 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:35.267 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:35.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:35.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:36.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:36.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:36.424 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:36.424 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:36.424 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:36.425 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:36.425 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:36.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:36.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:37.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:37.584 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:37.585 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:37.585 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:37.585 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:37.585 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:37.585 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:37.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:37.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:38.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:38.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:38.745 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:38.745 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:38.745 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:38.745 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:38.746 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:38.906 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:38.906 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:39.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:39.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:39.908 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:39.908 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:39.908 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:39.908 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:39.909 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:40.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:40.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:41.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:41.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:41.071 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:41.071 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:41.071 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:41.071 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:41.072 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:41.230 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:41.230 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:42.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:42.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:42.232 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:42.232 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:42.233 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:42.233 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:42.233 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:42.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:42.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:43.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:43.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:43.390 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:43.390 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:43.390 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:43.390 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:43.390 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:43.548 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:43.548 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:44.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:44.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:44.550 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:44.550 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:44.551 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:44.551 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:44.551 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:44.707 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:44.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:45.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:45.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:45.710 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:45.710 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:45.710 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:45.710 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:45.710 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:45.869 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:45.870 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:46.871 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:46.871 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:46.872 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:46.872 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:46.873 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:46.873 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:46.873 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:47.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:47.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:48.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:48.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:48.036 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:48.037 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:48.037 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:48.037 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:48.037 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:48.198 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:48.199 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:49.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:49.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:49.200 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:49.201 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:49.201 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:49.201 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:49.202 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:49.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:49.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:50.361 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:50.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:50.362 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:50.363 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:50.363 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:50.363 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:50.363 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:50.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:50.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:51.523 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:51.523 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:51.524 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:51.524 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:51.524 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:51.524 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:51.524 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:51.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:51.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:52.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:52.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:52.689 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:52.689 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:52.689 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:52.690 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:52.690 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:52.848 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:52.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:53.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:53.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:53.849 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:53.849 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:53.849 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:53.850 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:53.850 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:54.006 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:54.006 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:55.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:55.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:55.008 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:55.008 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:55.009 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:55.009 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:55.009 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:55.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:55.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:56.170 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:56.170 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:56.171 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:56.171 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:56.171 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:56.171 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:56.171 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:56.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:56.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:57.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:57.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:57.329 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:57.329 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:57.330 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:57.330 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:57.330 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:57.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:57.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:58.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:58.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:58.490 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:58.490 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:58.490 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:58.490 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:58.491 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:58.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:58.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:37:59.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:37:59.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:37:59.651 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:37:59.651 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:37:59.651 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:37:59.651 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:37:59.651 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:37:59.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:37:59.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:38:00.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:38:00.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:38:00.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:38:00.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:38:00.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:38:00.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:38:00.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:38:00.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:38:00.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:38:01.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:38:01.970 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:38:01.971 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:38:01.971 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:38:01.971 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:38:01.971 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:38:01.971 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:38:02.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:38:02.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:38:03.128 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:38:03.128 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:38:03.129 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:38:03.129 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:38:03.129 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:38:03.130 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:38:03.130 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:38:03.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:38:03.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:38:04.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:38:04.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:38:04.288 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:38:04.288 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:38:04.288 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:38:04.288 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:38:04.289 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:38:04.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:38:04.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:38:05.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:38:05.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:38:05.448 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:38:05.448 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:38:05.448 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:38:05.448 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:38:05.448 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:38:05.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:38:05.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:38:06.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:38:06.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:38:06.605 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:38:06.605 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:38:06.605 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:38:06.605 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:38:06.605 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:38:06.764 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:38:06.764 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:38:07.766 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:38:07.766 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:38:07.766 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:38:07.767 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:38:07.767 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:38:07.767 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:38:07.767 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:38:07.924 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:38:07.924 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:38:08.925 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:38:08.926 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:38:08.926 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:38:08.926 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:38:08.926 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:38:08.926 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:38:08.927 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:38:09.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:38:09.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:38:10.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:38:10.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:38:10.083 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:38:10.083 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:38:10.083 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:38:10.084 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:38:10.084 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:38:10.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:38:10.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:38:11.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:38:11.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:38:11.246 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:38:11.246 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:38:11.246 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:38:11.246 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:38:11.246 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:38:11.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:38:11.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:38:12.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:38:12.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:38:12.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:38:12.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:38:12.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:38:12.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:38:12.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:38:12.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:38:12.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:38:13.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:38:13.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:38:13.567 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:38:13.567 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:38:13.567 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:38:13.568 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:38:13.568 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:38:13.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:38:13.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:38:14.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:38:14.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:38:14.728 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:38:14.729 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:38:14.729 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:38:14.729 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:38:14.729 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:38:14.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:38:14.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:38:15.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:38:15.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:38:15.889 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:38:15.889 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:38:15.889 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:38:15.889 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:38:15.890 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:38:16.048 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:38:16.049 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:38:17.049 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:38:17.049 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:38:17.050 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:38:17.050 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:38:17.050 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:38:17.050 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:38:17.050 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:38:17.210 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:38:17.210 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:38:18.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:38:18.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:38:18.211 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:38:18.211 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:38:18.211 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:38:18.212 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:38:18.212 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:38:18.372 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:38:18.372 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:38:19.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:38:19.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:38:19.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:38:19.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:38:19.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:38:19.375 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:38:19.375 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:38:19.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:38:19.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:38:20.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:38:20.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:38:20.532 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:38:20.532 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:38:20.532 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:38:20.532 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:38:20.532 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:38:20.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:38:20.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:38:21.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:38:21.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:38:21.688 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:38:21.688 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:38:21.688 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:38:21.688 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:38:21.689 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:38:21.845 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:38:21.845 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:38:22.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:38:22.847 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:38:22.847 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:38:22.847 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:38:22.848 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:38:22.848 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:38:22.848 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:38:23.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:38:23.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:38:24.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:38:24.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:38:24.011 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:38:24.011 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:38:24.011 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:38:24.011 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:38:24.012 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:38:24.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:38:24.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:38:25.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:38:25.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:38:25.173 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:38:25.173 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:38:25.173 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:38:25.173 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:38:25.173 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:38:25.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:38:25.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:38:26.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:38:26.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:38:26.335 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:38:26.336 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:38:26.336 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:38:26.336 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:38:26.336 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:38:26.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:38:26.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:38:27.494 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:38:27.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:38:27.496 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:38:27.496 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:38:27.496 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:38:27.496 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:38:27.496 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:38:27.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:38:27.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:38:28.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:38:28.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:38:28.658 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:38:28.658 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:38:28.658 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:38:28.658 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:38:28.658 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:38:28.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:38:28.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:38:29.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:38:29.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:38:29.820 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:38:29.820 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:38:29.820 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:38:29.820 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:38:29.820 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:38:29.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:38:29.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:38:30.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:38:30.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:38:30.978 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:38:30.978 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:38:30.978 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:38:30.978 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:38:30.978 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:38:31.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:38:31.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:38:32.136 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:38:32.136 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:38:32.137 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:38:32.138 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:38:32.138 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:38:32.138 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:38:32.138 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:38:32.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:38:32.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:38:33.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:38:33.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:38:33.302 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:38:33.303 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:38:33.303 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:38:33.303 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:38:33.303 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:38:33.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:38:33.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:38:34.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:38:34.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:38:34.469 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:38:34.469 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:38:34.469 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:38:34.470 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:38:34.470 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:38:34.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:38:34.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:38:35.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:38:35.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:38:35.632 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:38:35.632 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:38:35.632 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:38:35.633 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:38:35.633 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:38:35.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:38:35.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:38:36.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:38:36.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:38:36.795 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:38:36.795 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:38:36.795 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:38:36.796 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:38:36.796 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:38:36.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:29:42.203241+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:38:36.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:38:37.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:38:37.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:38:37.957 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:38:37.957 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:38:37.957 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:38:37.957 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:38:37.958 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:38:38.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:38:36.561852+0000 '>' 2026-02-13T02:29:42.203241+0000 2026-02-13T02:38:38.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:38:38.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:400: _scrub_abort: perf_counters td/osd-scrub-test 3 2026-02-13T02:38:38.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2026-02-13T02:38:38.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=3 2026-02-13T02:38:38.116 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 3 - 1 2026-02-13T02:38:38.116 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 2 2026-02-13T02:38:38.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:38:38.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2026-02-13T02:38:38.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-13T02:38:38.197 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:38:38.197 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_ec": [ 2026-02-13T02:38:38.197 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:38:38.197 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:38:38.198 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:38:38.198 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:38:38.198 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:38:38.198 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:38:38.198 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:38:38.198 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:38:38.198 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:38:38.198 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:38:38.198 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:38:38.198 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:38:38.198 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:38:38.198 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:38:38.198 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_repl": [ 2026-02-13T02:38:38.198 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:38:38.198 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:38:38.198 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:38:38.198 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:38:38.198 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:38:38.198 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:38:38.198 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:38:38.198 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:38:38.198 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:38:38.198 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:38:38.198 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:38:38.198 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:38:38.199 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:38:38.199 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:38:38.199 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_ec": [ 2026-02-13T02:38:38.199 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:38:38.199 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:38:38.199 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:38:38.199 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:38:38.199 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:38:38.199 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:38:38.199 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:38:38.199 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:38:38.199 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:38:38.199 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:38:38.199 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:38:38.199 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:38:38.199 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:38:38.199 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:38:38.199 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_repl": [ 2026-02-13T02:38:38.199 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:38:38.199 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:38:38.200 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:38:38.200 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:38:38.200 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:38:38.200 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:38:38.200 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:38:38.200 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:38:38.200 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:38:38.200 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:38:38.200 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:38:38.200 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:38:38.200 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:38:38.200 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:38:38.200 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:38:38.201 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:38:38.201 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2026-02-13T02:38:38.201 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-13T02:38:38.271 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:38:38.271 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_ec": [ 2026-02-13T02:38:38.271 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:38:38.271 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:38:38.271 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:38:38.271 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:38:38.271 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:38:38.271 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:38:38.272 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:38:38.272 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:38:38.272 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:38:38.272 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:38:38.272 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:38:38.272 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:38:38.272 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:38:38.272 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:38:38.272 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_repl": [ 2026-02-13T02:38:38.272 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:38:38.272 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:38:38.272 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:38:38.272 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:38:38.272 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:38:38.272 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:38:38.272 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:38:38.272 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 67, 2026-02-13T02:38:38.272 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:38:38.272 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:38:38.272 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:38:38.272 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:38:38.272 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:38:38.272 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:38:38.272 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_ec": [ 2026-02-13T02:38:38.272 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:38:38.272 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:38:38.272 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:38:38.272 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:38:38.272 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:38:38.272 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:38:38.272 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:38:38.272 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:38:38.272 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:38:38.272 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:38:38.272 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:38:38.272 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:38:38.272 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:38:38.273 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:38:38.273 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_repl": [ 2026-02-13T02:38:38.273 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:38:38.273 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:38:38.273 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:38:38.273 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:38:38.273 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:38:38.273 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:38:38.273 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:38:38.273 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:38:38.273 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:38:38.273 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:38:38.273 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:38:38.273 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:38:38.273 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:38:38.273 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:38:38.273 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:38:38.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:38:38.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.2 counter dump 2026-02-13T02:38:38.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-13T02:38:38.342 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:38:38.342 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_ec": [ 2026-02-13T02:38:38.342 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:38:38.342 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:38:38.342 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:38:38.342 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:38:38.343 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:38:38.343 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:38:38.343 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:38:38.343 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:38:38.343 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:38:38.343 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:38:38.343 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:38:38.343 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:38:38.343 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:38:38.343 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:38:38.343 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_repl": [ 2026-02-13T02:38:38.343 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:38:38.344 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:38:38.344 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:38:38.344 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:38:38.344 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:38:38.344 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:38:38.344 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:38:38.344 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:38:38.344 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:38:38.344 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:38:38.344 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:38:38.344 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:38:38.344 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:38:38.344 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:38:38.345 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_ec": [ 2026-02-13T02:38:38.345 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:38:38.345 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:38:38.345 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:38:38.345 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:38:38.345 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:38:38.345 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:38:38.345 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:38:38.345 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:38:38.345 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:38:38.345 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:38:38.345 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:38:38.346 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:38:38.346 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:38:38.346 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:38:38.346 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_repl": [ 2026-02-13T02:38:38.346 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:38:38.346 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:38:38.346 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:38:38.346 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:38:38.346 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:38:38.346 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:38:38.346 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:38:38.346 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:38:38.346 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:38:38.347 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:38:38.347 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:38:38.347 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:38:38.347 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:38:38.347 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:38:38.347 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:38:38.347 INFO:tasks.workunit.client.0.trial112.stdout:-------------- Teardown Test TEST_deep_scrub_abort ------------------ 2026-02-13T02:38:38.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_deep_scrub_abort ------------------' 2026-02-13T02:38:38.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2026-02-13T02:38:38.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-13T02:38:38.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:38:38.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-13T02:38:38.349 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:38:38.349 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:38:38.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:38:38.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:38:38.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:38:38.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:38:38.452 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:38:38.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:38:38.453 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:38:38.455 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:38:38.455 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:38:38.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:38:38.456 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:38:38.456 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:38:38.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:38:38.457 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:38:38.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:38:38.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:38:38.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-13T02:38:38.518 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:38:38.518 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:38:38.518 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:38:38.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.508946 2026-02-13T02:38:38.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:38:38.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:38:38.520 INFO:tasks.workunit.client.0.trial112.stdout:-------------- Complete Test TEST_deep_scrub_abort ------------------ 2026-02-13T02:38:38.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_deep_scrub_abort ------------------' 2026-02-13T02:38:38.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2026-02-13T02:38:38.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_dump_scrub_schedule -------------------' 2026-02-13T02:38:38.521 INFO:tasks.workunit.client.0.trial112.stdout:-------------- Prepare Test TEST_dump_scrub_schedule ------------------- 2026-02-13T02:38:38.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2026-02-13T02:38:38.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2026-02-13T02:38:38.523 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2026-02-13T02:38:38.523 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-13T02:38:38.523 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:38:38.523 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-13T02:38:38.523 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:38:38.523 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:38:38.523 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:38:38.523 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:38:38.523 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:38:38.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:38:38.524 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:38:38.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:38:38.525 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:38:38.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:38:38.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:38:38.527 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:38:38.527 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:38:38.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:38:38.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:38:38.528 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:38:38.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:38:38.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:38:38.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-13T02:38:38.532 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:38:38.532 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:38:38.532 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:38:38.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.508946 2026-02-13T02:38:38.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:38:38.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:38:38.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2026-02-13T02:38:38.535 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T02:38:38.535 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:38:38.535 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:38:38.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.508946 2026-02-13T02:38:38.536 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T02:38:38.537 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-13T02:38:38.537 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T02:38:38.537 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2026-02-13T02:38:38.538 INFO:tasks.workunit.client.0.trial112.stdout:-------------- Run Test TEST_dump_scrub_schedule ----------------------- 2026-02-13T02:38:38.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_dump_scrub_schedule -----------------------' 2026-02-13T02:38:38.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_dump_scrub_schedule td/osd-scrub-test 2026-02-13T02:38:38.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:530: TEST_dump_scrub_schedule: local dir=td/osd-scrub-test 2026-02-13T02:38:38.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:531: TEST_dump_scrub_schedule: local poolname=test 2026-02-13T02:38:38.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:532: TEST_dump_scrub_schedule: local OSDS=3 2026-02-13T02:38:38.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:533: TEST_dump_scrub_schedule: local objects=90 2026-02-13T02:38:38.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:535: TEST_dump_scrub_schedule: TESTDATA=testdata.508946 2026-02-13T02:38:38.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:537: TEST_dump_scrub_schedule: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2026-02-13T02:38:38.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-test 2026-02-13T02:38:38.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-13T02:38:38.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-13T02:38:38.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-13T02:38:38.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-test/a 2026-02-13T02:38:38.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2026-02-13T02:38:38.559 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-13T02:38:38.559 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:38:38.559 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:38:38.559 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:38:38.559 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:38:38.559 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:38:38.560 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:38:38.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2026-02-13T02:38:38.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-13T02:38:38.588 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-13T02:38:38.588 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:38:38.588 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:38:38.589 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-13T02:38:38.590 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-13T02:38:38.590 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:38:38.590 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:38:38.590 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:38:38.590 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:38:38.590 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:38:38.590 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:38:38.590 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.508946/ceph-mon.a.asok 2026-02-13T02:38:38.591 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:38:38.591 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.508946/ceph-mon.a.asok config get fsid 2026-02-13T02:38:38.659 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-13T02:38:38.660 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:38:38.660 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:38:38.660 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-13T02:38:38.661 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-13T02:38:38.661 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:38:38.661 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:38:38.661 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:38:38.661 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:38:38.661 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:38:38.662 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:38:38.662 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.508946/ceph-mon.a.asok 2026-02-13T02:38:38.662 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:38:38.662 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.508946/ceph-mon.a.asok config get mon_host 2026-02-13T02:38:38.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:538: TEST_dump_scrub_schedule: run_mgr td/osd-scrub-test x --mgr_stats_period=1 2026-02-13T02:38:38.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-test 2026-02-13T02:38:38.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-13T02:38:38.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-13T02:38:38.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-13T02:38:38.718 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-test/x 2026-02-13T02:38:38.718 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-13T02:38:38.810 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-13T02:38:38.810 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:38:38.810 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:38:38.811 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:38:38.811 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:38:38.811 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:38:38.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:38:38.812 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:38:38.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mgr_stats_period=1 2026-02-13T02:38:38.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:552: TEST_dump_scrub_schedule: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=1 --osd_scrub_retry_pg_state=2 --osd_scrub_retry_delay=2 --osd_scrub_sleep=0.2' 2026-02-13T02:38:38.831 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:554: TEST_dump_scrub_schedule: expr 3 - 1 2026-02-13T02:38:38.832 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:554: TEST_dump_scrub_schedule: seq 0 2 2026-02-13T02:38:38.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:554: TEST_dump_scrub_schedule: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:38:38.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:556: TEST_dump_scrub_schedule: run_osd td/osd-scrub-test 0 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=1 --osd_scrub_retry_pg_state=2 --osd_scrub_retry_delay=2 --osd_scrub_sleep=0.2 2026-02-13T02:38:38.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-13T02:38:38.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:38:38.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-13T02:38:38.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:38:38.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/0 2026-02-13T02:38:38.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-13T02:38:38.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:38:38.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:38:38.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:38:38.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2026-02-13T02:38:38.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2026-02-13T02:38:38.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:38:38.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:38:38.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-13T02:38:38.834 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:38:38.834 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:38:38.834 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:38:38.834 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:38:38.834 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:38:38.834 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:38:38.836 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:38:38.836 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:38:38.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:38:38.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:38:38.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:38:38.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-13T02:38:38.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-13T02:38:38.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:38:38.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:38:38.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:38:38.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:38:38.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:38:38.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=1 --osd_scrub_retry_pg_state=2 --osd_scrub_retry_delay=2 --osd_scrub_sleep=0.2' 2026-02-13T02:38:38.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/0 2026-02-13T02:38:38.838 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:38:38.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=61664d51-438d-4040-acd2-b015322b7bb9 2026-02-13T02:38:38.838 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 61664d51-438d-4040-acd2-b015322b7bb9 2026-02-13T02:38:38.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 61664d51-438d-4040-acd2-b015322b7bb9' 2026-02-13T02:38:38.839 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:38:38.847 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCujo5pQ9ouMhAAU/mNmYQrJRXI5DVYfpqi2A== 2026-02-13T02:38:38.847 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCujo5pQ9ouMhAAU/mNmYQrJRXI5DVYfpqi2A=="}' 2026-02-13T02:38:38.847 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 61664d51-438d-4040-acd2-b015322b7bb9 -i td/osd-scrub-test/0/new.json 2026-02-13T02:38:38.933 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:38:38.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/0/new.json 2026-02-13T02:38:38.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=1 --osd_scrub_retry_pg_state=2 --osd_scrub_retry_delay=2 --osd_scrub_sleep=0.2 --mkfs --key AQCujo5pQ9ouMhAAU/mNmYQrJRXI5DVYfpqi2A== --osd-uuid 61664d51-438d-4040-acd2-b015322b7bb9 2026-02-13T02:38:38.960 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:38:38.951+0000 7f62f85c9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:38:38.961 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:38:38.951+0000 7f62f85c9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:38:38.962 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:38:38.951+0000 7f62f85c9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:38:38.963 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:38:38.951+0000 7f62f85c9a40 -1 bdev(0x55a2da018c00 td/osd-scrub-test/0/block) open stat got: (1) Operation not permitted 2026-02-13T02:38:38.963 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:38:38.951+0000 7f62f85c9a40 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2026-02-13T02:38:39.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2026-02-13T02:38:39.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:38:39.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-13T02:38:39.664 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T02:38:39.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:38:39.770 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-13T02:38:39.771 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T02:38:39.771 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=1 --osd_scrub_retry_pg_state=2 --osd_scrub_retry_delay=2 --osd_scrub_sleep=0.2 2026-02-13T02:38:39.771 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:38:39.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:38:39.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:38:39.800 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:38:39.791+0000 7f9858d9aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:38:39.802 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:38:39.791+0000 7f9858d9aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:38:39.803 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:38:39.795+0000 7f9858d9aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:38:39.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-13T02:38:39.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:38:39.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:38:39.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:38:39.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:38:39.866 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:38:39.866 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:38:39.866 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:38:39.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:38:39.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:38:39.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:38:40.177 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:38:40.167+0000 7f9858d9aa40 -1 Falling back to public interface 2026-02-13T02:38:40.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:38:40.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:38:40.957 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:38:40.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:38:40.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:38:40.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:38:41.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:38:41.234 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:38:41.223+0000 7f9858d9aa40 -1 osd.0 0 log_to_monitors true 2026-02-13T02:38:42.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:38:42.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:38:42.083 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:38:42.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:38:42.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:38:42.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:38:42.245 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1624705642,v1:127.0.0.1:6801/1624705642] [v2:127.0.0.1:6802/1624705642,v1:127.0.0.1:6803/1624705642] exists,up 61664d51-438d-4040-acd2-b015322b7bb9 2026-02-13T02:38:42.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:38:42.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:38:42.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:38:42.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:554: TEST_dump_scrub_schedule: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:38:42.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:556: TEST_dump_scrub_schedule: run_osd td/osd-scrub-test 1 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=1 --osd_scrub_retry_pg_state=2 --osd_scrub_retry_delay=2 --osd_scrub_sleep=0.2 2026-02-13T02:38:42.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-13T02:38:42.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:38:42.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-13T02:38:42.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:38:42.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/1 2026-02-13T02:38:42.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-13T02:38:42.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:38:42.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:38:42.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:38:42.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2026-02-13T02:38:42.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2026-02-13T02:38:42.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:38:42.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:38:42.251 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-13T02:38:42.251 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:38:42.251 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:38:42.251 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:38:42.251 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:38:42.251 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:38:42.251 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:38:42.251 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:38:42.251 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:38:42.251 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:38:42.251 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:38:42.251 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:38:42.251 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-13T02:38:42.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-13T02:38:42.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:38:42.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:38:42.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:38:42.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:38:42.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:38:42.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=1 --osd_scrub_retry_pg_state=2 --osd_scrub_retry_delay=2 --osd_scrub_sleep=0.2' 2026-02-13T02:38:42.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/1 2026-02-13T02:38:42.253 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:38:42.253 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ac590a3c-85a3-4b35-a357-d0e7ed820f6f 2026-02-13T02:38:42.253 INFO:tasks.workunit.client.0.trial112.stdout:add osd1 ac590a3c-85a3-4b35-a357-d0e7ed820f6f 2026-02-13T02:38:42.254 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 ac590a3c-85a3-4b35-a357-d0e7ed820f6f' 2026-02-13T02:38:42.254 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:38:42.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCyjo5p9ZRRDxAA5G29HbkxEjgeQhxdTQkCuA== 2026-02-13T02:38:42.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCyjo5p9ZRRDxAA5G29HbkxEjgeQhxdTQkCuA=="}' 2026-02-13T02:38:42.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ac590a3c-85a3-4b35-a357-d0e7ed820f6f -i td/osd-scrub-test/1/new.json 2026-02-13T02:38:42.426 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:38:42.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/1/new.json 2026-02-13T02:38:42.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=1 --osd_scrub_retry_pg_state=2 --osd_scrub_retry_delay=2 --osd_scrub_sleep=0.2 --mkfs --key AQCyjo5p9ZRRDxAA5G29HbkxEjgeQhxdTQkCuA== --osd-uuid ac590a3c-85a3-4b35-a357-d0e7ed820f6f 2026-02-13T02:38:42.455 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:38:42.447+0000 7fbf9df30a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:38:42.457 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:38:42.447+0000 7fbf9df30a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:38:42.458 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:38:42.447+0000 7fbf9df30a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:38:42.458 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:38:42.447+0000 7fbf9df30a40 -1 bdev(0x5634c6deac00 td/osd-scrub-test/1/block) open stat got: (1) Operation not permitted 2026-02-13T02:38:42.459 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:38:42.447+0000 7fbf9df30a40 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2026-02-13T02:38:43.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2026-02-13T02:38:43.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:38:43.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-13T02:38:43.084 INFO:tasks.workunit.client.0.trial112.stdout:adding osd1 key to auth repository 2026-02-13T02:38:43.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:38:43.291 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T02:38:43.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-13T02:38:43.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=1 --osd_scrub_retry_pg_state=2 --osd_scrub_retry_delay=2 --osd_scrub_sleep=0.2 2026-02-13T02:38:43.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:38:43.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:38:43.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:38:43.318 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:38:43.307+0000 7f81c3c09a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:38:43.320 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:38:43.311+0000 7f81c3c09a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:38:43.320 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:38:43.311+0000 7f81c3c09a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:38:43.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-13T02:38:43.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:38:43.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:38:43.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:38:43.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:38:43.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:38:43.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:38:43.454 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:38:43.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:38:43.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:38:43.613 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:38:43.705 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:38:43.695+0000 7f81c3c09a40 -1 Falling back to public interface 2026-02-13T02:38:44.294 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:38:44.283+0000 7f81c3c09a40 -1 osd.1 0 log_to_monitors true 2026-02-13T02:38:44.615 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:38:44.615 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:38:44.615 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:38:44.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:38:44.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:38:44.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:38:44.776 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1919479818,v1:127.0.0.1:6811/1919479818] [v2:127.0.0.1:6812/1919479818,v1:127.0.0.1:6813/1919479818] exists,up ac590a3c-85a3-4b35-a357-d0e7ed820f6f 2026-02-13T02:38:44.779 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:38:44.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:38:44.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:38:44.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:554: TEST_dump_scrub_schedule: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:38:44.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:556: TEST_dump_scrub_schedule: run_osd td/osd-scrub-test 2 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=1 --osd_scrub_retry_pg_state=2 --osd_scrub_retry_delay=2 --osd_scrub_sleep=0.2 2026-02-13T02:38:44.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-13T02:38:44.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:38:44.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-13T02:38:44.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:38:44.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/2 2026-02-13T02:38:44.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-13T02:38:44.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:38:44.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:38:44.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:38:44.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2026-02-13T02:38:44.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2026-02-13T02:38:44.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:38:44.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:38:44.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-13T02:38:44.781 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:38:44.781 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:38:44.781 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:38:44.781 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:38:44.781 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:38:44.782 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:38:44.782 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:38:44.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:38:44.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:38:44.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:38:44.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:38:44.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-13T02:38:44.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-13T02:38:44.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:38:44.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:38:44.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:38:44.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:38:44.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:38:44.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=1 --osd_scrub_retry_pg_state=2 --osd_scrub_retry_delay=2 --osd_scrub_sleep=0.2' 2026-02-13T02:38:44.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/2 2026-02-13T02:38:44.783 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:38:44.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=756fb250-36c8-45c4-a391-8afb893d28ec 2026-02-13T02:38:44.784 INFO:tasks.workunit.client.0.trial112.stdout:add osd2 756fb250-36c8-45c4-a391-8afb893d28ec 2026-02-13T02:38:44.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 756fb250-36c8-45c4-a391-8afb893d28ec' 2026-02-13T02:38:44.784 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:38:44.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC0jo5p14HzLhAATDAvFKKDkCCoFVtwclg9Wg== 2026-02-13T02:38:44.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC0jo5p14HzLhAATDAvFKKDkCCoFVtwclg9Wg=="}' 2026-02-13T02:38:44.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 756fb250-36c8-45c4-a391-8afb893d28ec -i td/osd-scrub-test/2/new.json 2026-02-13T02:38:44.944 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:38:44.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/2/new.json 2026-02-13T02:38:44.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=1 --osd_scrub_retry_pg_state=2 --osd_scrub_retry_delay=2 --osd_scrub_sleep=0.2 --mkfs --key AQC0jo5p14HzLhAATDAvFKKDkCCoFVtwclg9Wg== --osd-uuid 756fb250-36c8-45c4-a391-8afb893d28ec 2026-02-13T02:38:44.973 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:38:44.963+0000 7f81f02d8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:38:44.975 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:38:44.963+0000 7f81f02d8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:38:44.976 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:38:44.967+0000 7f81f02d8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:38:44.977 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:38:44.967+0000 7f81f02d8a40 -1 bdev(0x557dd1926c00 td/osd-scrub-test/2/block) open stat got: (1) Operation not permitted 2026-02-13T02:38:44.977 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:38:44.967+0000 7f81f02d8a40 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2026-02-13T02:38:45.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2026-02-13T02:38:45.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:38:45.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-13T02:38:45.631 INFO:tasks.workunit.client.0.trial112.stdout:adding osd2 key to auth repository 2026-02-13T02:38:45.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:38:45.838 INFO:tasks.workunit.client.0.trial112.stdout:start osd.2 2026-02-13T02:38:45.839 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-13T02:38:45.839 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=1 --osd_scrub_retry_pg_state=2 --osd_scrub_retry_delay=2 --osd_scrub_sleep=0.2 2026-02-13T02:38:45.839 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:38:45.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:38:45.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:38:45.866 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:38:45.855+0000 7feb9a536a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:38:45.868 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:38:45.859+0000 7feb9a536a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:38:45.869 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:38:45.859+0000 7feb9a536a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:38:45.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-13T02:38:46.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:38:46.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T02:38:46.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:38:46.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:38:46.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:38:46.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:38:46.000 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:38:46.001 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:38:46.001 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:38:46.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:38:46.257 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:38:46.247+0000 7feb9a536a40 -1 Falling back to public interface 2026-02-13T02:38:46.901 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:38:46.891+0000 7feb9a536a40 -1 osd.2 0 log_to_monitors true 2026-02-13T02:38:47.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:38:47.160 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:38:47.160 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:38:47.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:38:47.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:38:47.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:38:47.316 INFO:tasks.workunit.client.0.trial112.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2221084049,v1:127.0.0.1:6819/2221084049] [v2:127.0.0.1:6820/2221084049,v1:127.0.0.1:6821/2221084049] exists,up 756fb250-36c8-45c4-a391-8afb893d28ec 2026-02-13T02:38:47.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:38:47.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:38:47.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:38:47.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: create_pool test 1 1 2026-02-13T02:38:47.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2026-02-13T02:38:47.522 INFO:tasks.workunit.client.0.trial112.stderr:pool 'test' created 2026-02-13T02:38:47.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:38:48.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:561: TEST_dump_scrub_schedule: wait_for_clean 2026-02-13T02:38:48.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:38:48.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:38:48.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:38:48.535 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:38:48.535 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:38:48.535 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:38:48.536 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:38:48.536 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:38:48.536 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:38:48.643 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:38:48.643 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:38:48.643 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:38:48.643 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:38:48.643 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:38:48.643 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:38:48.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:38:48.805 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:38:48.805 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:38:48.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:38:48.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:38:48.806 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:38:48.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836488 2026-02-13T02:38:48.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836488 2026-02-13T02:38:48.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488' 2026-02-13T02:38:48.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:38:48.880 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:38:48.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705669 2026-02-13T02:38:48.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705669 2026-02-13T02:38:48.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-38654705669' 2026-02-13T02:38:48.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:38:48.956 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:38:49.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574850 2026-02-13T02:38:49.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574850 2026-02-13T02:38:49.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-38654705669 2-55834574850' 2026-02-13T02:38:49.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:38:49.031 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836488 2026-02-13T02:38:49.031 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:38:49.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:38:49.033 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836488 2026-02-13T02:38:49.033 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:38:49.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836488 2026-02-13T02:38:49.034 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836488 2026-02-13T02:38:49.035 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2026-02-13T02:38:49.035 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:38:49.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836488 2026-02-13T02:38:49.201 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:38:49.201 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705669 2026-02-13T02:38:49.201 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:38:49.202 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:38:49.203 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705669 2026-02-13T02:38:49.203 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:38:49.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705669 2026-02-13T02:38:49.205 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705669 2026-02-13T02:38:49.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705669' 2026-02-13T02:38:49.206 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:38:49.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705669 -lt 38654705669 2026-02-13T02:38:49.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:38:49.371 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574850 2026-02-13T02:38:49.371 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:38:49.372 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:38:49.373 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574850 2026-02-13T02:38:49.373 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:38:49.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574850 2026-02-13T02:38:49.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574850' 2026-02-13T02:38:49.375 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574850 2026-02-13T02:38:49.375 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:38:49.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574850 -lt 55834574850 2026-02-13T02:38:49.545 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:38:49.545 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:38:49.545 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:38:49.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:38:49.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:38:49.737 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:38:49.737 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:38:49.737 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:38:49.737 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:38:49.738 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:38:49.738 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:38:49.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:38:49.891 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:38:49.892 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:38:49.892 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:38:50.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:38:50.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:38:50.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:38:50.082 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: ceph osd dump 2026-02-13T02:38:50.082 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: grep '^pool.*['\'']test['\'']' 2026-02-13T02:38:50.082 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: awk '{ print $2 }' 2026-02-13T02:38:50.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: poolid=1 2026-02-13T02:38:50.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:564: TEST_dump_scrub_schedule: dd if=/dev/urandom of=testdata.508946 bs=1032 count=1 2026-02-13T02:38:50.241 INFO:tasks.workunit.client.0.trial112.stderr:1+0 records in 2026-02-13T02:38:50.241 INFO:tasks.workunit.client.0.trial112.stderr:1+0 records out 2026-02-13T02:38:50.241 INFO:tasks.workunit.client.0.trial112.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 5.7939e-05 s, 17.8 MB/s 2026-02-13T02:38:50.242 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: seq 1 90 2026-02-13T02:38:50.243 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:50.243 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj1 testdata.508946 2026-02-13T02:38:50.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:50.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj2 testdata.508946 2026-02-13T02:38:50.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:50.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj3 testdata.508946 2026-02-13T02:38:50.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:50.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj4 testdata.508946 2026-02-13T02:38:50.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:50.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj5 testdata.508946 2026-02-13T02:38:50.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:50.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj6 testdata.508946 2026-02-13T02:38:50.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:50.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj7 testdata.508946 2026-02-13T02:38:50.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:50.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj8 testdata.508946 2026-02-13T02:38:50.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:50.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj9 testdata.508946 2026-02-13T02:38:50.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:50.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj10 testdata.508946 2026-02-13T02:38:50.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:50.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj11 testdata.508946 2026-02-13T02:38:50.511 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:50.511 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj12 testdata.508946 2026-02-13T02:38:50.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:50.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj13 testdata.508946 2026-02-13T02:38:50.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:50.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj14 testdata.508946 2026-02-13T02:38:50.585 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:50.585 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj15 testdata.508946 2026-02-13T02:38:50.610 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:50.610 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj16 testdata.508946 2026-02-13T02:38:50.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:50.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj17 testdata.508946 2026-02-13T02:38:50.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:50.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj18 testdata.508946 2026-02-13T02:38:50.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:50.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj19 testdata.508946 2026-02-13T02:38:50.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:50.697 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj20 testdata.508946 2026-02-13T02:38:50.720 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:50.720 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj21 testdata.508946 2026-02-13T02:38:50.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:50.745 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj22 testdata.508946 2026-02-13T02:38:50.769 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:50.769 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj23 testdata.508946 2026-02-13T02:38:50.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:50.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj24 testdata.508946 2026-02-13T02:38:50.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:50.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj25 testdata.508946 2026-02-13T02:38:50.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:50.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj26 testdata.508946 2026-02-13T02:38:50.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:50.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj27 testdata.508946 2026-02-13T02:38:50.893 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:50.893 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj28 testdata.508946 2026-02-13T02:38:50.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:50.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj29 testdata.508946 2026-02-13T02:38:50.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:50.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj30 testdata.508946 2026-02-13T02:38:50.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:50.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj31 testdata.508946 2026-02-13T02:38:50.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:50.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj32 testdata.508946 2026-02-13T02:38:51.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj33 testdata.508946 2026-02-13T02:38:51.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj34 testdata.508946 2026-02-13T02:38:51.056 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.056 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj35 testdata.508946 2026-02-13T02:38:51.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj36 testdata.508946 2026-02-13T02:38:51.105 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.105 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj37 testdata.508946 2026-02-13T02:38:51.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj38 testdata.508946 2026-02-13T02:38:51.155 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.155 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj39 testdata.508946 2026-02-13T02:38:51.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj40 testdata.508946 2026-02-13T02:38:51.203 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj41 testdata.508946 2026-02-13T02:38:51.229 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.229 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj42 testdata.508946 2026-02-13T02:38:51.253 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.254 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj43 testdata.508946 2026-02-13T02:38:51.271 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.271 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj44 testdata.508946 2026-02-13T02:38:51.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj45 testdata.508946 2026-02-13T02:38:51.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj46 testdata.508946 2026-02-13T02:38:51.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj47 testdata.508946 2026-02-13T02:38:51.359 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.359 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj48 testdata.508946 2026-02-13T02:38:51.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj49 testdata.508946 2026-02-13T02:38:51.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj50 testdata.508946 2026-02-13T02:38:51.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj51 testdata.508946 2026-02-13T02:38:51.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj52 testdata.508946 2026-02-13T02:38:51.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj53 testdata.508946 2026-02-13T02:38:51.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj54 testdata.508946 2026-02-13T02:38:51.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj55 testdata.508946 2026-02-13T02:38:51.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj56 testdata.508946 2026-02-13T02:38:51.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj57 testdata.508946 2026-02-13T02:38:51.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj58 testdata.508946 2026-02-13T02:38:51.619 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.620 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj59 testdata.508946 2026-02-13T02:38:51.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj60 testdata.508946 2026-02-13T02:38:51.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj61 testdata.508946 2026-02-13T02:38:51.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj62 testdata.508946 2026-02-13T02:38:51.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj63 testdata.508946 2026-02-13T02:38:51.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj64 testdata.508946 2026-02-13T02:38:51.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj65 testdata.508946 2026-02-13T02:38:51.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj66 testdata.508946 2026-02-13T02:38:51.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj67 testdata.508946 2026-02-13T02:38:51.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj68 testdata.508946 2026-02-13T02:38:51.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj69 testdata.508946 2026-02-13T02:38:51.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj70 testdata.508946 2026-02-13T02:38:51.898 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.898 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj71 testdata.508946 2026-02-13T02:38:51.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj72 testdata.508946 2026-02-13T02:38:51.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj73 testdata.508946 2026-02-13T02:38:51.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj74 testdata.508946 2026-02-13T02:38:51.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:51.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj75 testdata.508946 2026-02-13T02:38:52.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:52.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj76 testdata.508946 2026-02-13T02:38:52.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:52.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj77 testdata.508946 2026-02-13T02:38:52.054 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:52.054 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj78 testdata.508946 2026-02-13T02:38:52.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:52.078 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj79 testdata.508946 2026-02-13T02:38:52.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:52.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj80 testdata.508946 2026-02-13T02:38:52.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:52.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj81 testdata.508946 2026-02-13T02:38:52.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:52.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj82 testdata.508946 2026-02-13T02:38:52.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:52.170 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj83 testdata.508946 2026-02-13T02:38:52.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:52.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj84 testdata.508946 2026-02-13T02:38:52.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:52.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj85 testdata.508946 2026-02-13T02:38:52.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:52.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj86 testdata.508946 2026-02-13T02:38:52.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:52.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj87 testdata.508946 2026-02-13T02:38:52.290 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:52.290 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj88 testdata.508946 2026-02-13T02:38:52.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:52.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj89 testdata.508946 2026-02-13T02:38:52.339 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-13T02:38:52.339 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj90 testdata.508946 2026-02-13T02:38:52.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:569: TEST_dump_scrub_schedule: rm -f testdata.508946 2026-02-13T02:38:52.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:571: TEST_dump_scrub_schedule: local pgid=1.0 2026-02-13T02:38:52.365 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:573: TEST_dump_scrub_schedule: date +%Y-%m-%dT%H:%M:%S.%N%:z 2026-02-13T02:38:52.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:573: TEST_dump_scrub_schedule: local now_is=2026-02-13T02:38:52.361385812+00:00 2026-02-13T02:38:52.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:581: TEST_dump_scrub_schedule: expct_starting=(['query_active']='false' ['query_is_future']='true' ['query_schedule']='scrub scheduled') 2026-02-13T02:38:52.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:581: TEST_dump_scrub_schedule: declare -A expct_starting 2026-02-13T02:38:52.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:582: TEST_dump_scrub_schedule: declare -A sched_data 2026-02-13T02:38:52.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:583: TEST_dump_scrub_schedule: extract_published_sch 1.0 2026-02-13T02:38:52.361385812+00:00 2019-10-12T20:32:43.645168+0000 sched_data 2026-02-13T02:38:52.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:17: extract_published_sch: local pgn=1.0 2026-02-13T02:38:52.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:18: extract_published_sch: local -n dict=sched_data 2026-02-13T02:38:52.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:19: extract_published_sch: local current_time=2026-02-13T02:38:52.361385812+00:00 2026-02-13T02:38:52.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:20: extract_published_sch: local extra_time=2019-10-12T20:32:43.645168+0000 2026-02-13T02:38:52.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:21: extract_published_sch: local extr_dbg=2 2026-02-13T02:38:52.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:24: extract_published_sch: local saved_echo_flag=x 2026-02-13T02:38:52.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:25: extract_published_sch: set +x 2026-02-13T02:38:52.514 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:38:52.526 INFO:tasks.workunit.client.0.trial112.stdout:from pg dump pg: { 2026-02-13T02:38:52.526 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_pg_state": "active+clean", 2026-02-13T02:38:52.526 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_state_has_scrubbing": false, 2026-02-13T02:38:52.527 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_last_duration": 0, 2026-02-13T02:38:52.527 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_schedule": "periodic scrub scheduled", 2026-02-13T02:38:52.527 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_schedule_at": "2026-02-14T02:38:47.516500+0000", 2026-02-13T02:38:52.527 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_is_future": true, 2026-02-13T02:38:52.527 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_vs_date": true, 2026-02-13T02:38:52.527 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_reported_epoch": 17, 2026-02-13T02:38:52.527 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_seq": 26 2026-02-13T02:38:52.527 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:38:52.527 INFO:tasks.workunit.client.0.trial112.stdout:query output: 2026-02-13T02:38:52.600 INFO:tasks.workunit.client.0.trial112.stdout: "scrubber": { 2026-02-13T02:38:52.600 INFO:tasks.workunit.client.0.trial112.stdout: "active": false, 2026-02-13T02:38:52.600 INFO:tasks.workunit.client.0.trial112.stdout: "must_scrub": false, 2026-02-13T02:38:52.600 INFO:tasks.workunit.client.0.trial112.stdout: "must_deep_scrub": false, 2026-02-13T02:38:52.600 INFO:tasks.workunit.client.0.trial112.stdout: "must_repair": false, 2026-02-13T02:38:52.601 INFO:tasks.workunit.client.0.trial112.stdout: "scrub_reg_stamp": "2026-02-14T02:38:47.516500+0000", 2026-02-13T02:38:52.601 INFO:tasks.workunit.client.0.trial112.stdout: "schedule": "scrub scheduled @ 2026-02-14T02:38:47.516 (2026-02-14T02:38:47.516)" 2026-02-13T02:38:52.601 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:38:52.601 INFO:tasks.workunit.client.0.trial112.stdout: "agent_state": {} 2026-02-13T02:38:52.684 INFO:tasks.workunit.client.0.trial112.stdout:( 2026-02-13T02:38:52.684 INFO:tasks.workunit.client.0.trial112.stdout:[query_epoch]=17 2026-02-13T02:38:52.685 INFO:tasks.workunit.client.0.trial112.stdout:[query_seq]=108 2026-02-13T02:38:52.685 INFO:tasks.workunit.client.0.trial112.stdout:[query_active]=false 2026-02-13T02:38:52.685 INFO:tasks.workunit.client.0.trial112.stdout:[query_schedule]='scrub scheduled' 2026-02-13T02:38:52.685 INFO:tasks.workunit.client.0.trial112.stdout:[query_schedule_at]='2026-02-14T02:38:47.516' 2026-02-13T02:38:52.685 INFO:tasks.workunit.client.0.trial112.stdout:[query_target_at]='2026-02-14T02:38:47.516' 2026-02-13T02:38:52.685 INFO:tasks.workunit.client.0.trial112.stdout:[query_last_duration]=0 2026-02-13T02:38:52.685 INFO:tasks.workunit.client.0.trial112.stdout:[query_last_stamp]='2026-02-13T02:38:47.516500+0000' 2026-02-13T02:38:52.685 INFO:tasks.workunit.client.0.trial112.stdout:[query_last_scrub]='0x0' 2026-02-13T02:38:52.685 INFO:tasks.workunit.client.0.trial112.stdout:[query_is_future]=true 2026-02-13T02:38:52.685 INFO:tasks.workunit.client.0.trial112.stdout:[query_vs_date]=true 2026-02-13T02:38:52.685 INFO:tasks.workunit.client.0.trial112.stdout:[query_scrub_seq]=null 2026-02-13T02:38:52.686 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_pg_state]='active+clean' 2026-02-13T02:38:52.686 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_state_has_scrubbing]=false 2026-02-13T02:38:52.686 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_last_duration]=0 2026-02-13T02:38:52.686 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_schedule]='periodic scrub scheduled' 2026-02-13T02:38:52.686 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_schedule_at]='2026-02-14T02:38:47.516500+0000' 2026-02-13T02:38:52.686 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_is_future]=true 2026-02-13T02:38:52.686 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_vs_date]=true 2026-02-13T02:38:52.686 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_reported_epoch]=17 2026-02-13T02:38:52.686 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_seq]=26 2026-02-13T02:38:52.686 INFO:tasks.workunit.client.0.trial112.stdout:) 2026-02-13T02:38:52.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:584: TEST_dump_scrub_schedule: schedule_against_expected sched_data expct_starting initial 2026-02-13T02:38:52.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:178: schedule_against_expected: local -n dict=sched_data 2026-02-13T02:38:52.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:179: schedule_against_expected: local -n ep=expct_starting 2026-02-13T02:38:52.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:180: schedule_against_expected: local extr_dbg=1 2026-02-13T02:38:52.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:183: schedule_against_expected: local saved_echo_flag=x 2026-02-13T02:38:52.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:184: schedule_against_expected: set +x 2026-02-13T02:38:52.697 INFO:tasks.workunit.client.0.trial112.stdout:-- - comparing: 2026-02-13T02:38:52.697 INFO:tasks.workunit.client.0.trial112.stdout:key is query_schedule expected: scrub scheduled in actual: scrub scheduled 2026-02-13T02:38:52.697 INFO:tasks.workunit.client.0.trial112.stdout:key is query_is_future expected: true in actual: true 2026-02-13T02:38:52.697 INFO:tasks.workunit.client.0.trial112.stdout:key is query_active expected: false in actual: false 2026-02-13T02:38:52.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:203: schedule_against_expected: return 0 2026-02-13T02:38:52.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:585: TEST_dump_scrub_schedule: (( 0 == 0 )) 2026-02-13T02:38:52.698 INFO:tasks.workunit.client.0.trial112.stdout:last-scrub --- 0x0 2026-02-13T02:38:52.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:586: TEST_dump_scrub_schedule: echo 'last-scrub --- ' 0x0 2026-02-13T02:38:52.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:592: TEST_dump_scrub_schedule: saved_last_stamp=2026-02-13T02:38:47.516500+0000 2026-02-13T02:38:52.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:593: TEST_dump_scrub_schedule: ceph tell 'osd.*' config set osd_scrub_sleep 0 2026-02-13T02:38:52.759 INFO:tasks.workunit.client.0.trial112.stdout:osd.0: { 2026-02-13T02:38:52.759 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_sleep = '' " 2026-02-13T02:38:52.759 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:38:52.767 INFO:tasks.workunit.client.0.trial112.stdout:osd.1: { 2026-02-13T02:38:52.767 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_sleep = '' " 2026-02-13T02:38:52.767 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:38:52.774 INFO:tasks.workunit.client.0.trial112.stdout:osd.2: { 2026-02-13T02:38:52.775 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_sleep = '' " 2026-02-13T02:38:52.775 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:38:52.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:594: TEST_dump_scrub_schedule: ceph tell 1.0 deep-scrub 2026-02-13T02:38:52.858 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:38:52.858 INFO:tasks.workunit.client.0.trial112.stdout: "deep": true, 2026-02-13T02:38:52.858 INFO:tasks.workunit.client.0.trial112.stdout: "must": true, 2026-02-13T02:38:52.858 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "0.000000" 2026-02-13T02:38:52.858 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:38:52.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:599: TEST_dump_scrub_schedule: sleep 5 2026-02-13T02:38:57.866 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:600: TEST_dump_scrub_schedule: sched_data=() 2026-02-13T02:38:57.866 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:601: TEST_dump_scrub_schedule: expct_qry_duration=(['query_last_duration']='0' ['query_last_duration_neg']='not0') 2026-02-13T02:38:57.866 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:601: TEST_dump_scrub_schedule: declare -A expct_qry_duration 2026-02-13T02:38:57.866 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:602: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2026-02-13T02:38:47.516500+0000 expct_qry_duration 'WaitingAfterScrub ' sched_data 2026-02-13T02:38:57.866 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:106: wait_any_cond: local pgid=1.0 2026-02-13T02:38:57.866 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local retries=10 2026-02-13T02:38:57.866 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: local cmp_date=2026-02-13T02:38:47.516500+0000 2026-02-13T02:38:57.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:109: wait_any_cond: local -n ep=expct_qry_duration 2026-02-13T02:38:57.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:110: wait_any_cond: local -n out_array=sched_data 2026-02-13T02:38:57.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:111: wait_any_cond: local -A sc_data 2026-02-13T02:38:57.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:112: wait_any_cond: local extr_dbg=2 2026-02-13T02:38:57.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:115: wait_any_cond: local saved_echo_flag=x 2026-02-13T02:38:57.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:116: wait_any_cond: set +x 2026-02-13T02:38:57.868 INFO:tasks.workunit.client.0.trial112.stdout:waiting for any condition (WaitingAfterScrub ): pg:1.0 dt:2026-02-13T02:38:47.516500+0000 (10 retries) 2026-02-13T02:38:58.523 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:38:58.532 INFO:tasks.workunit.client.0.trial112.stdout:from pg dump pg: { 2026-02-13T02:38:58.532 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_pg_state": "active+clean", 2026-02-13T02:38:58.533 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_state_has_scrubbing": false, 2026-02-13T02:38:58.533 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_last_duration": 1, 2026-02-13T02:38:58.533 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_schedule": "periodic scrub scheduled", 2026-02-13T02:38:58.533 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_schedule_at": "2026-02-14T02:38:53.382955+0000", 2026-02-13T02:38:58.533 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_is_future": true, 2026-02-13T02:38:58.533 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_vs_date": true, 2026-02-13T02:38:58.533 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_reported_epoch": 17, 2026-02-13T02:38:58.533 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_seq": 124 2026-02-13T02:38:58.533 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:38:58.533 INFO:tasks.workunit.client.0.trial112.stdout:query output: 2026-02-13T02:38:58.609 INFO:tasks.workunit.client.0.trial112.stdout: "scrubber": { 2026-02-13T02:38:58.609 INFO:tasks.workunit.client.0.trial112.stdout: "active": false, 2026-02-13T02:38:58.609 INFO:tasks.workunit.client.0.trial112.stdout: "must_scrub": false, 2026-02-13T02:38:58.609 INFO:tasks.workunit.client.0.trial112.stdout: "must_deep_scrub": false, 2026-02-13T02:38:58.609 INFO:tasks.workunit.client.0.trial112.stdout: "must_repair": false, 2026-02-13T02:38:58.609 INFO:tasks.workunit.client.0.trial112.stdout: "scrub_reg_stamp": "2026-02-14T02:38:53.382955+0000", 2026-02-13T02:38:58.609 INFO:tasks.workunit.client.0.trial112.stdout: "schedule": "scrub scheduled @ 2026-02-14T02:38:53.382 (2026-02-14T02:38:53.382)" 2026-02-13T02:38:58.609 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:38:58.609 INFO:tasks.workunit.client.0.trial112.stdout: "agent_state": {} 2026-02-13T02:38:58.691 INFO:tasks.workunit.client.0.trial112.stdout:( 2026-02-13T02:38:58.691 INFO:tasks.workunit.client.0.trial112.stdout:[query_epoch]=17 2026-02-13T02:38:58.691 INFO:tasks.workunit.client.0.trial112.stdout:[query_seq]=126 2026-02-13T02:38:58.691 INFO:tasks.workunit.client.0.trial112.stdout:[query_active]=false 2026-02-13T02:38:58.691 INFO:tasks.workunit.client.0.trial112.stdout:[query_schedule]='scrub scheduled' 2026-02-13T02:38:58.691 INFO:tasks.workunit.client.0.trial112.stdout:[query_schedule_at]='2026-02-14T02:38:53.382' 2026-02-13T02:38:58.691 INFO:tasks.workunit.client.0.trial112.stdout:[query_target_at]='2026-02-14T02:38:53.382' 2026-02-13T02:38:58.691 INFO:tasks.workunit.client.0.trial112.stdout:[query_last_duration]=1 2026-02-13T02:38:58.691 INFO:tasks.workunit.client.0.trial112.stdout:[query_last_stamp]='2026-02-13T02:38:53.382955+0000' 2026-02-13T02:38:58.691 INFO:tasks.workunit.client.0.trial112.stdout:[query_last_scrub]='17x90' 2026-02-13T02:38:58.691 INFO:tasks.workunit.client.0.trial112.stdout:[query_is_future]=true 2026-02-13T02:38:58.691 INFO:tasks.workunit.client.0.trial112.stdout:[query_vs_date]=true 2026-02-13T02:38:58.691 INFO:tasks.workunit.client.0.trial112.stdout:[query_scrub_seq]=null 2026-02-13T02:38:58.691 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_pg_state]='active+clean' 2026-02-13T02:38:58.691 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_state_has_scrubbing]=false 2026-02-13T02:38:58.691 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_last_duration]=1 2026-02-13T02:38:58.691 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_schedule]='periodic scrub scheduled' 2026-02-13T02:38:58.691 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_schedule_at]='2026-02-14T02:38:53.382955+0000' 2026-02-13T02:38:58.691 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_is_future]=true 2026-02-13T02:38:58.691 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_vs_date]=true 2026-02-13T02:38:58.691 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_reported_epoch]=17 2026-02-13T02:38:58.691 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_seq]=124 2026-02-13T02:38:58.691 INFO:tasks.workunit.client.0.trial112.stdout:) 2026-02-13T02:38:58.702 INFO:tasks.workunit.client.0.trial112.stdout:--> loop: 1 ~ false / 126 / 124 / true / 2026-02-13T02:38:53.382955+0000 / scrub scheduled %%% query_last_duration_neg query_last_duration 2026-02-13T02:38:58.702 INFO:tasks.workunit.client.0.trial112.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 1 2026-02-13T02:38:58.702 INFO:tasks.workunit.client.0.trial112.stdout:WaitingAfterScrub - 'query_last_duration' actual value (1) matches expected (0) (negation: 1) 2026-02-13T02:38:58.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:157: wait_any_cond: return 0 2026-02-13T02:38:58.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:604: TEST_dump_scrub_schedule: sched_data=() 2026-02-13T02:38:58.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:605: TEST_dump_scrub_schedule: expct_dmp_duration=(['dmp_last_duration']='0' ['dmp_last_duration_neg']='not0') 2026-02-13T02:38:58.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:605: TEST_dump_scrub_schedule: declare -A expct_dmp_duration 2026-02-13T02:38:58.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:606: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2026-02-13T02:38:47.516500+0000 expct_dmp_duration 'WaitingAfterScrub_dmp ' sched_data 2026-02-13T02:38:58.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:106: wait_any_cond: local pgid=1.0 2026-02-13T02:38:58.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local retries=10 2026-02-13T02:38:58.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: local cmp_date=2026-02-13T02:38:47.516500+0000 2026-02-13T02:38:58.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:109: wait_any_cond: local -n ep=expct_dmp_duration 2026-02-13T02:38:58.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:110: wait_any_cond: local -n out_array=sched_data 2026-02-13T02:38:58.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:111: wait_any_cond: local -A sc_data 2026-02-13T02:38:58.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:112: wait_any_cond: local extr_dbg=2 2026-02-13T02:38:58.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:115: wait_any_cond: local saved_echo_flag=x 2026-02-13T02:38:58.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:116: wait_any_cond: set +x 2026-02-13T02:38:58.704 INFO:tasks.workunit.client.0.trial112.stdout:waiting for any condition (WaitingAfterScrub_dmp ): pg:1.0 dt:2026-02-13T02:38:47.516500+0000 (10 retries) 2026-02-13T02:38:59.360 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:38:59.370 INFO:tasks.workunit.client.0.trial112.stdout:from pg dump pg: { 2026-02-13T02:38:59.370 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_pg_state": "active+clean", 2026-02-13T02:38:59.370 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_state_has_scrubbing": false, 2026-02-13T02:38:59.370 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_last_duration": 1, 2026-02-13T02:38:59.370 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_schedule": "periodic scrub scheduled", 2026-02-13T02:38:59.370 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_schedule_at": "2026-02-14T02:38:53.382955+0000", 2026-02-13T02:38:59.370 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_is_future": true, 2026-02-13T02:38:59.370 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_vs_date": true, 2026-02-13T02:38:59.370 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_reported_epoch": 17, 2026-02-13T02:38:59.370 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_seq": 125 2026-02-13T02:38:59.370 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:38:59.370 INFO:tasks.workunit.client.0.trial112.stdout:query output: 2026-02-13T02:38:59.442 INFO:tasks.workunit.client.0.trial112.stdout: "scrubber": { 2026-02-13T02:38:59.443 INFO:tasks.workunit.client.0.trial112.stdout: "active": false, 2026-02-13T02:38:59.443 INFO:tasks.workunit.client.0.trial112.stdout: "must_scrub": false, 2026-02-13T02:38:59.443 INFO:tasks.workunit.client.0.trial112.stdout: "must_deep_scrub": false, 2026-02-13T02:38:59.443 INFO:tasks.workunit.client.0.trial112.stdout: "must_repair": false, 2026-02-13T02:38:59.443 INFO:tasks.workunit.client.0.trial112.stdout: "scrub_reg_stamp": "2026-02-14T02:38:53.382955+0000", 2026-02-13T02:38:59.443 INFO:tasks.workunit.client.0.trial112.stdout: "schedule": "scrub scheduled @ 2026-02-14T02:38:53.382 (2026-02-14T02:38:53.382)" 2026-02-13T02:38:59.443 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:38:59.443 INFO:tasks.workunit.client.0.trial112.stdout: "agent_state": {} 2026-02-13T02:38:59.530 INFO:tasks.workunit.client.0.trial112.stdout:( 2026-02-13T02:38:59.530 INFO:tasks.workunit.client.0.trial112.stdout:[query_epoch]=17 2026-02-13T02:38:59.530 INFO:tasks.workunit.client.0.trial112.stdout:[query_seq]=126 2026-02-13T02:38:59.530 INFO:tasks.workunit.client.0.trial112.stdout:[query_active]=false 2026-02-13T02:38:59.530 INFO:tasks.workunit.client.0.trial112.stdout:[query_schedule]='scrub scheduled' 2026-02-13T02:38:59.530 INFO:tasks.workunit.client.0.trial112.stdout:[query_schedule_at]='2026-02-14T02:38:53.382' 2026-02-13T02:38:59.530 INFO:tasks.workunit.client.0.trial112.stdout:[query_target_at]='2026-02-14T02:38:53.382' 2026-02-13T02:38:59.530 INFO:tasks.workunit.client.0.trial112.stdout:[query_last_duration]=1 2026-02-13T02:38:59.530 INFO:tasks.workunit.client.0.trial112.stdout:[query_last_stamp]='2026-02-13T02:38:53.382955+0000' 2026-02-13T02:38:59.530 INFO:tasks.workunit.client.0.trial112.stdout:[query_last_scrub]='17x90' 2026-02-13T02:38:59.530 INFO:tasks.workunit.client.0.trial112.stdout:[query_is_future]=true 2026-02-13T02:38:59.530 INFO:tasks.workunit.client.0.trial112.stdout:[query_vs_date]=true 2026-02-13T02:38:59.530 INFO:tasks.workunit.client.0.trial112.stdout:[query_scrub_seq]=null 2026-02-13T02:38:59.530 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_pg_state]='active+clean' 2026-02-13T02:38:59.530 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_state_has_scrubbing]=false 2026-02-13T02:38:59.530 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_last_duration]=1 2026-02-13T02:38:59.530 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_schedule]='periodic scrub scheduled' 2026-02-13T02:38:59.530 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_schedule_at]='2026-02-14T02:38:53.382955+0000' 2026-02-13T02:38:59.530 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_is_future]=true 2026-02-13T02:38:59.530 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_vs_date]=true 2026-02-13T02:38:59.530 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_reported_epoch]=17 2026-02-13T02:38:59.530 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_seq]=125 2026-02-13T02:38:59.530 INFO:tasks.workunit.client.0.trial112.stdout:) 2026-02-13T02:38:59.540 INFO:tasks.workunit.client.0.trial112.stdout:--> loop: 1 ~ false / 126 / 125 / true / 2026-02-13T02:38:53.382955+0000 / scrub scheduled %%% dmp_last_duration dmp_last_duration_neg 2026-02-13T02:38:59.540 INFO:tasks.workunit.client.0.trial112.stdout:key is dmp_last_duration: negation:1 # expected: 0 # in actual: 1 2026-02-13T02:38:59.540 INFO:tasks.workunit.client.0.trial112.stdout:WaitingAfterScrub_dmp - 'dmp_last_duration' actual value (1) matches expected (0) (negation: 1) 2026-02-13T02:38:59.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:157: wait_any_cond: return 0 2026-02-13T02:38:59.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:613: TEST_dump_scrub_schedule: ceph osd set noscrub 2026-02-13T02:38:59.744 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:38:59.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:614: TEST_dump_scrub_schedule: sleep 2 2026-02-13T02:39:01.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:615: TEST_dump_scrub_schedule: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_max 3 2026-02-13T02:39:01.822 INFO:tasks.workunit.client.0.trial112.stdout:osd.0: { 2026-02-13T02:39:01.823 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_shallow_scrub_chunk_max = '' " 2026-02-13T02:39:01.823 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:39:01.830 INFO:tasks.workunit.client.0.trial112.stdout:osd.1: { 2026-02-13T02:39:01.830 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_shallow_scrub_chunk_max = '' " 2026-02-13T02:39:01.830 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:39:01.838 INFO:tasks.workunit.client.0.trial112.stdout:osd.2: { 2026-02-13T02:39:01.838 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_shallow_scrub_chunk_max = '' " 2026-02-13T02:39:01.838 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:39:01.845 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:616: TEST_dump_scrub_schedule: ceph tell 'osd.*' config set osd_scrub_sleep 2.0 2026-02-13T02:39:01.922 INFO:tasks.workunit.client.0.trial112.stdout:osd.0: { 2026-02-13T02:39:01.922 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_sleep = '' " 2026-02-13T02:39:01.922 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:39:01.929 INFO:tasks.workunit.client.0.trial112.stdout:osd.1: { 2026-02-13T02:39:01.929 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_sleep = '' " 2026-02-13T02:39:01.929 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:39:01.936 INFO:tasks.workunit.client.0.trial112.stdout:osd.2: { 2026-02-13T02:39:01.936 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_sleep = '' " 2026-02-13T02:39:01.937 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:39:01.944 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:617: TEST_dump_scrub_schedule: sleep 8 2026-02-13T02:39:09.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:618: TEST_dump_scrub_schedule: saved_last_stamp=2026-02-13T02:38:53.382955+0000 2026-02-13T02:39:09.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:620: TEST_dump_scrub_schedule: ceph tell 1.0 schedule-scrub 2026-02-13T02:39:10.011 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:39:10.012 INFO:tasks.workunit.client.0.trial112.stdout: "deep": false, 2026-02-13T02:39:10.012 INFO:tasks.workunit.client.0.trial112.stdout: "must": false, 2026-02-13T02:39:10.012 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "2026-02-11T02:39:10.007136+0000" 2026-02-13T02:39:10.012 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:39:10.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:621: TEST_dump_scrub_schedule: sleep 1 2026-02-13T02:39:11.021 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:622: TEST_dump_scrub_schedule: sched_data=() 2026-02-13T02:39:11.021 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:623: TEST_dump_scrub_schedule: expct_scrub_peri_sched=(['query_is_future']='false') 2026-02-13T02:39:11.021 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:623: TEST_dump_scrub_schedule: declare -A expct_scrub_peri_sched 2026-02-13T02:39:11.021 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:624: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2026-02-13T02:38:53.382955+0000 expct_scrub_peri_sched waitingBeingScheduled sched_data 2026-02-13T02:39:11.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:106: wait_any_cond: local pgid=1.0 2026-02-13T02:39:11.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local retries=10 2026-02-13T02:39:11.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: local cmp_date=2026-02-13T02:38:53.382955+0000 2026-02-13T02:39:11.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:109: wait_any_cond: local -n ep=expct_scrub_peri_sched 2026-02-13T02:39:11.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:110: wait_any_cond: local -n out_array=sched_data 2026-02-13T02:39:11.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:111: wait_any_cond: local -A sc_data 2026-02-13T02:39:11.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:112: wait_any_cond: local extr_dbg=2 2026-02-13T02:39:11.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:115: wait_any_cond: local saved_echo_flag=x 2026-02-13T02:39:11.022 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:116: wait_any_cond: set +x 2026-02-13T02:39:11.023 INFO:tasks.workunit.client.0.trial112.stdout:waiting for any condition (waitingBeingScheduled): pg:1.0 dt:2026-02-13T02:38:53.382955+0000 (10 retries) 2026-02-13T02:39:11.671 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:39:11.681 INFO:tasks.workunit.client.0.trial112.stdout:from pg dump pg: { 2026-02-13T02:39:11.681 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_pg_state": "active+clean", 2026-02-13T02:39:11.681 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_state_has_scrubbing": false, 2026-02-13T02:39:11.681 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_last_duration": 1, 2026-02-13T02:39:11.681 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_schedule": "periodic scrub scheduled", 2026-02-13T02:39:11.681 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_schedule_at": "2026-02-13T02:39:11.398242+0000", 2026-02-13T02:39:11.681 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_is_future": true, 2026-02-13T02:39:11.682 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_vs_date": true, 2026-02-13T02:39:11.682 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_reported_epoch": 18, 2026-02-13T02:39:11.682 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_seq": 135 2026-02-13T02:39:11.682 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:39:11.682 INFO:tasks.workunit.client.0.trial112.stdout:query output: 2026-02-13T02:39:11.757 INFO:tasks.workunit.client.0.trial112.stdout: "scrubber": { 2026-02-13T02:39:11.757 INFO:tasks.workunit.client.0.trial112.stdout: "active": false, 2026-02-13T02:39:11.757 INFO:tasks.workunit.client.0.trial112.stdout: "must_scrub": false, 2026-02-13T02:39:11.757 INFO:tasks.workunit.client.0.trial112.stdout: "must_deep_scrub": false, 2026-02-13T02:39:11.757 INFO:tasks.workunit.client.0.trial112.stdout: "must_repair": false, 2026-02-13T02:39:11.757 INFO:tasks.workunit.client.0.trial112.stdout: "scrub_reg_stamp": "2026-02-13T02:39:12.444816+0000", 2026-02-13T02:39:11.757 INFO:tasks.workunit.client.0.trial112.stdout: "schedule": "scrub scheduled @ 2026-02-13T02:39:12.444 (2026-02-12T02:39:10.007)" 2026-02-13T02:39:11.757 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:39:11.758 INFO:tasks.workunit.client.0.trial112.stdout: "agent_state": {} 2026-02-13T02:39:11.838 INFO:tasks.workunit.client.0.trial112.stdout:( 2026-02-13T02:39:11.839 INFO:tasks.workunit.client.0.trial112.stdout:[query_epoch]=18 2026-02-13T02:39:11.839 INFO:tasks.workunit.client.0.trial112.stdout:[query_seq]=136 2026-02-13T02:39:11.839 INFO:tasks.workunit.client.0.trial112.stdout:[query_active]=false 2026-02-13T02:39:11.839 INFO:tasks.workunit.client.0.trial112.stdout:[query_schedule]='scrub scheduled' 2026-02-13T02:39:11.839 INFO:tasks.workunit.client.0.trial112.stdout:[query_schedule_at]='2026-02-13T02:39:12.444' 2026-02-13T02:39:11.839 INFO:tasks.workunit.client.0.trial112.stdout:[query_target_at]='2026-02-12T02:39:10.007' 2026-02-13T02:39:11.839 INFO:tasks.workunit.client.0.trial112.stdout:[query_last_duration]=1 2026-02-13T02:39:11.839 INFO:tasks.workunit.client.0.trial112.stdout:[query_last_stamp]='2026-02-11T02:39:10.007136+0000' 2026-02-13T02:39:11.839 INFO:tasks.workunit.client.0.trial112.stdout:[query_last_scrub]='17x90' 2026-02-13T02:39:11.839 INFO:tasks.workunit.client.0.trial112.stdout:[query_is_future]=false 2026-02-13T02:39:11.839 INFO:tasks.workunit.client.0.trial112.stdout:[query_vs_date]=true 2026-02-13T02:39:11.839 INFO:tasks.workunit.client.0.trial112.stdout:[query_scrub_seq]=null 2026-02-13T02:39:11.839 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_pg_state]='active+clean' 2026-02-13T02:39:11.839 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_state_has_scrubbing]=false 2026-02-13T02:39:11.839 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_last_duration]=1 2026-02-13T02:39:11.839 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_schedule]='periodic scrub scheduled' 2026-02-13T02:39:11.839 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_schedule_at]='2026-02-13T02:39:11.398242+0000' 2026-02-13T02:39:11.839 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_is_future]=true 2026-02-13T02:39:11.839 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_vs_date]=true 2026-02-13T02:39:11.840 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_reported_epoch]=18 2026-02-13T02:39:11.840 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_seq]=135 2026-02-13T02:39:11.840 INFO:tasks.workunit.client.0.trial112.stdout:) 2026-02-13T02:39:11.850 INFO:tasks.workunit.client.0.trial112.stdout:--> loop: 1 ~ false / 136 / 135 / false / 2026-02-11T02:39:10.007136+0000 / scrub scheduled %%% query_is_future 2026-02-13T02:39:11.851 INFO:tasks.workunit.client.0.trial112.stdout:key is query_is_future: negation:0 # expected: false # in actual: false 2026-02-13T02:39:11.851 INFO:tasks.workunit.client.0.trial112.stdout:waitingBeingScheduled - 'query_is_future' actual value (false) matches expected (false) (negation: 0) 2026-02-13T02:39:11.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:157: wait_any_cond: return 0 2026-02-13T02:39:11.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:636: TEST_dump_scrub_schedule: saved_last_stamp=2026-02-11T02:39:10.007136+0000 2026-02-13T02:39:11.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:637: TEST_dump_scrub_schedule: ceph osd unset noscrub 2026-02-13T02:39:12.053 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is unset 2026-02-13T02:39:12.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:639: TEST_dump_scrub_schedule: cond_active=(['query_active']='true') 2026-02-13T02:39:12.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:639: TEST_dump_scrub_schedule: declare -A cond_active 2026-02-13T02:39:12.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:640: TEST_dump_scrub_schedule: sched_data=() 2026-02-13T02:39:12.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:641: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2026-02-11T02:39:10.007136+0000 cond_active 'WaitingActive ' sched_data 2026-02-13T02:39:12.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:106: wait_any_cond: local pgid=1.0 2026-02-13T02:39:12.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local retries=10 2026-02-13T02:39:12.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: local cmp_date=2026-02-11T02:39:10.007136+0000 2026-02-13T02:39:12.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:109: wait_any_cond: local -n ep=cond_active 2026-02-13T02:39:12.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:110: wait_any_cond: local -n out_array=sched_data 2026-02-13T02:39:12.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:111: wait_any_cond: local -A sc_data 2026-02-13T02:39:12.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:112: wait_any_cond: local extr_dbg=2 2026-02-13T02:39:12.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:115: wait_any_cond: local saved_echo_flag=x 2026-02-13T02:39:12.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:116: wait_any_cond: set +x 2026-02-13T02:39:12.068 INFO:tasks.workunit.client.0.trial112.stdout:waiting for any condition (WaitingActive ): pg:1.0 dt:2026-02-11T02:39:10.007136+0000 (10 retries) 2026-02-13T02:39:12.721 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:39:12.731 INFO:tasks.workunit.client.0.trial112.stdout:from pg dump pg: { 2026-02-13T02:39:12.731 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_pg_state": "active+clean", 2026-02-13T02:39:12.731 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_state_has_scrubbing": false, 2026-02-13T02:39:12.731 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_last_duration": 1, 2026-02-13T02:39:12.731 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_schedule": "periodic scrub scheduled", 2026-02-13T02:39:12.731 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_schedule_at": "2026-02-13T02:39:11.398242+0000", 2026-02-13T02:39:12.731 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_is_future": false, 2026-02-13T02:39:12.731 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_vs_date": true, 2026-02-13T02:39:12.731 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_reported_epoch": 18, 2026-02-13T02:39:12.731 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_seq": 135 2026-02-13T02:39:12.731 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:39:12.731 INFO:tasks.workunit.client.0.trial112.stdout:query output: 2026-02-13T02:39:12.802 INFO:tasks.workunit.client.0.trial112.stdout: "scrubber": { 2026-02-13T02:39:12.802 INFO:tasks.workunit.client.0.trial112.stdout: "active": false, 2026-02-13T02:39:12.803 INFO:tasks.workunit.client.0.trial112.stdout: "must_scrub": false, 2026-02-13T02:39:12.803 INFO:tasks.workunit.client.0.trial112.stdout: "must_deep_scrub": false, 2026-02-13T02:39:12.803 INFO:tasks.workunit.client.0.trial112.stdout: "must_repair": false, 2026-02-13T02:39:12.803 INFO:tasks.workunit.client.0.trial112.stdout: "scrub_reg_stamp": "2026-02-13T02:39:12.444816+0000", 2026-02-13T02:39:12.803 INFO:tasks.workunit.client.0.trial112.stdout: "schedule": "queued for scrub at 2026-02-12T02:39:10.007" 2026-02-13T02:39:12.803 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:39:12.803 INFO:tasks.workunit.client.0.trial112.stdout: "agent_state": {} 2026-02-13T02:39:12.890 INFO:tasks.workunit.client.0.trial112.stdout:( 2026-02-13T02:39:12.890 INFO:tasks.workunit.client.0.trial112.stdout:[query_epoch]=19 2026-02-13T02:39:12.890 INFO:tasks.workunit.client.0.trial112.stdout:[query_seq]=139 2026-02-13T02:39:12.891 INFO:tasks.workunit.client.0.trial112.stdout:[query_active]=false 2026-02-13T02:39:12.891 INFO:tasks.workunit.client.0.trial112.stdout:[query_schedule]='queued for scrub at 2026-02-12T02:39:10.007' 2026-02-13T02:39:12.891 INFO:tasks.workunit.client.0.trial112.stdout:[query_schedule_at]='0' 2026-02-13T02:39:12.891 INFO:tasks.workunit.client.0.trial112.stdout:[query_target_at]='0' 2026-02-13T02:39:12.891 INFO:tasks.workunit.client.0.trial112.stdout:[query_last_duration]=1 2026-02-13T02:39:12.891 INFO:tasks.workunit.client.0.trial112.stdout:[query_last_stamp]='2026-02-11T02:39:10.007136+0000' 2026-02-13T02:39:12.891 INFO:tasks.workunit.client.0.trial112.stdout:[query_last_scrub]='17x90' 2026-02-13T02:39:12.891 INFO:tasks.workunit.client.0.trial112.stdout:[query_is_future]=false 2026-02-13T02:39:12.891 INFO:tasks.workunit.client.0.trial112.stdout:[query_vs_date]=false 2026-02-13T02:39:12.891 INFO:tasks.workunit.client.0.trial112.stdout:[query_scrub_seq]=null 2026-02-13T02:39:12.891 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_pg_state]='active+clean' 2026-02-13T02:39:12.891 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_state_has_scrubbing]=false 2026-02-13T02:39:12.891 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_last_duration]=1 2026-02-13T02:39:12.891 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_schedule]='periodic scrub scheduled' 2026-02-13T02:39:12.891 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_schedule_at]='2026-02-13T02:39:11.398242+0000' 2026-02-13T02:39:12.891 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_is_future]=false 2026-02-13T02:39:12.891 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_vs_date]=true 2026-02-13T02:39:12.891 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_reported_epoch]=18 2026-02-13T02:39:12.891 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_seq]=135 2026-02-13T02:39:12.891 INFO:tasks.workunit.client.0.trial112.stdout:) 2026-02-13T02:39:12.902 INFO:tasks.workunit.client.0.trial112.stdout:--> loop: 1 ~ false / 139 / 135 / false / 2026-02-11T02:39:10.007136+0000 / queued for scrub at 2026-02-12T02:39:10.007 %%% query_active 2026-02-13T02:39:12.902 INFO:tasks.workunit.client.0.trial112.stdout:key is query_active: negation:0 # expected: true # in actual: false 2026-02-13T02:39:13.557 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:39:13.567 INFO:tasks.workunit.client.0.trial112.stdout:from pg dump pg: { 2026-02-13T02:39:13.567 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_pg_state": "active+clean", 2026-02-13T02:39:13.567 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_state_has_scrubbing": false, 2026-02-13T02:39:13.567 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_last_duration": 1, 2026-02-13T02:39:13.567 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_schedule": "queued for scrub", 2026-02-13T02:39:13.568 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_schedule_at": "0", 2026-02-13T02:39:13.568 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_is_future": false, 2026-02-13T02:39:13.568 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_vs_date": false, 2026-02-13T02:39:13.568 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_reported_epoch": 19, 2026-02-13T02:39:13.568 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_seq": 138 2026-02-13T02:39:13.568 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:39:13.568 INFO:tasks.workunit.client.0.trial112.stdout:query output: 2026-02-13T02:39:13.643 INFO:tasks.workunit.client.0.trial112.stdout: "scrubber": { 2026-02-13T02:39:13.643 INFO:tasks.workunit.client.0.trial112.stdout: "active": true, 2026-02-13T02:39:13.643 INFO:tasks.workunit.client.0.trial112.stdout: "epoch_start": "14", 2026-02-13T02:39:13.643 INFO:tasks.workunit.client.0.trial112.stdout: "start": "1:00000000::::0", 2026-02-13T02:39:13.643 INFO:tasks.workunit.client.0.trial112.stdout: "end": "MIN", 2026-02-13T02:39:13.644 INFO:tasks.workunit.client.0.trial112.stdout: "max_end": "MIN", 2026-02-13T02:39:13.644 INFO:tasks.workunit.client.0.trial112.stdout: "subset_last_update": "0'0", 2026-02-13T02:39:13.644 INFO:tasks.workunit.client.0.trial112.stdout: "deep": false, 2026-02-13T02:39:13.644 INFO:tasks.workunit.client.0.trial112.stdout: "req_scrub": false, 2026-02-13T02:39:13.644 INFO:tasks.workunit.client.0.trial112.stdout: "auto_repair": false, 2026-02-13T02:39:13.644 INFO:tasks.workunit.client.0.trial112.stdout: "check_repair": false, 2026-02-13T02:39:13.644 INFO:tasks.workunit.client.0.trial112.stdout: "deep_scrub_on_error": false, 2026-02-13T02:39:13.644 INFO:tasks.workunit.client.0.trial112.stdout: "priority": 5, 2026-02-13T02:39:13.644 INFO:tasks.workunit.client.0.trial112.stdout: "shallow_errors": 0, 2026-02-13T02:39:13.644 INFO:tasks.workunit.client.0.trial112.stdout: "deep_errors": 0, 2026-02-13T02:39:13.644 INFO:tasks.workunit.client.0.trial112.stdout: "fixed": 0, 2026-02-13T02:39:13.645 INFO:tasks.workunit.client.0.trial112.stdout: "waiting_on_whom": [], 2026-02-13T02:39:13.645 INFO:tasks.workunit.client.0.trial112.stdout: "schedule": "scrubbing", 2026-02-13T02:39:13.645 INFO:tasks.workunit.client.0.trial112.stdout: "is_reserving_replicas": false 2026-02-13T02:39:13.645 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:39:13.645 INFO:tasks.workunit.client.0.trial112.stdout: "agent_state": {} 2026-02-13T02:39:13.728 INFO:tasks.workunit.client.0.trial112.stdout:( 2026-02-13T02:39:13.728 INFO:tasks.workunit.client.0.trial112.stdout:[query_epoch]=19 2026-02-13T02:39:13.728 INFO:tasks.workunit.client.0.trial112.stdout:[query_seq]=142 2026-02-13T02:39:13.728 INFO:tasks.workunit.client.0.trial112.stdout:[query_active]=true 2026-02-13T02:39:13.729 INFO:tasks.workunit.client.0.trial112.stdout:[query_schedule]='scrubbing' 2026-02-13T02:39:13.729 INFO:tasks.workunit.client.0.trial112.stdout:[query_schedule_at]='0' 2026-02-13T02:39:13.729 INFO:tasks.workunit.client.0.trial112.stdout:[query_target_at]='0' 2026-02-13T02:39:13.729 INFO:tasks.workunit.client.0.trial112.stdout:[query_last_duration]=1 2026-02-13T02:39:13.729 INFO:tasks.workunit.client.0.trial112.stdout:[query_last_stamp]='2026-02-11T02:39:10.007136+0000' 2026-02-13T02:39:13.729 INFO:tasks.workunit.client.0.trial112.stdout:[query_last_scrub]='17x90' 2026-02-13T02:39:13.729 INFO:tasks.workunit.client.0.trial112.stdout:[query_is_future]=false 2026-02-13T02:39:13.729 INFO:tasks.workunit.client.0.trial112.stdout:[query_vs_date]=false 2026-02-13T02:39:13.729 INFO:tasks.workunit.client.0.trial112.stdout:[query_scrub_seq]=null 2026-02-13T02:39:13.729 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_pg_state]='active+clean' 2026-02-13T02:39:13.730 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_state_has_scrubbing]=false 2026-02-13T02:39:13.730 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_last_duration]=1 2026-02-13T02:39:13.730 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_schedule]='queued for scrub' 2026-02-13T02:39:13.730 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_schedule_at]='0' 2026-02-13T02:39:13.730 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_is_future]=false 2026-02-13T02:39:13.730 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_vs_date]=false 2026-02-13T02:39:13.730 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_reported_epoch]=19 2026-02-13T02:39:13.730 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_seq]=138 2026-02-13T02:39:13.730 INFO:tasks.workunit.client.0.trial112.stdout:) 2026-02-13T02:39:13.740 INFO:tasks.workunit.client.0.trial112.stdout:--> loop: 2 ~ true / 142 / 138 / false / 2026-02-11T02:39:10.007136+0000 / scrubbing %%% query_active 2026-02-13T02:39:13.740 INFO:tasks.workunit.client.0.trial112.stdout:key is query_active: negation:0 # expected: true # in actual: true 2026-02-13T02:39:13.740 INFO:tasks.workunit.client.0.trial112.stdout:WaitingActive - 'query_active' actual value (true) matches expected (true) (negation: 0) 2026-02-13T02:39:13.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:157: wait_any_cond: return 0 2026-02-13T02:39:13.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:645: TEST_dump_scrub_schedule: cond_active_dmp=(['dmp_state_has_scrubbing']='true' ['query_active']='false') 2026-02-13T02:39:13.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:645: TEST_dump_scrub_schedule: declare -A cond_active_dmp 2026-02-13T02:39:13.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:646: TEST_dump_scrub_schedule: sched_data=() 2026-02-13T02:39:13.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:647: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2026-02-11T02:39:10.007136+0000 cond_active_dmp 'WaitingActive ' sched_data 2026-02-13T02:39:13.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:106: wait_any_cond: local pgid=1.0 2026-02-13T02:39:13.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local retries=10 2026-02-13T02:39:13.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: local cmp_date=2026-02-11T02:39:10.007136+0000 2026-02-13T02:39:13.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:109: wait_any_cond: local -n ep=cond_active_dmp 2026-02-13T02:39:13.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:110: wait_any_cond: local -n out_array=sched_data 2026-02-13T02:39:13.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:111: wait_any_cond: local -A sc_data 2026-02-13T02:39:13.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:112: wait_any_cond: local extr_dbg=2 2026-02-13T02:39:13.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:115: wait_any_cond: local saved_echo_flag=x 2026-02-13T02:39:13.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:116: wait_any_cond: set +x 2026-02-13T02:39:13.743 INFO:tasks.workunit.client.0.trial112.stdout:waiting for any condition (WaitingActive ): pg:1.0 dt:2026-02-11T02:39:10.007136+0000 (10 retries) 2026-02-13T02:39:14.393 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:39:14.402 INFO:tasks.workunit.client.0.trial112.stdout:from pg dump pg: { 2026-02-13T02:39:14.403 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_pg_state": "active+clean", 2026-02-13T02:39:14.403 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_state_has_scrubbing": false, 2026-02-13T02:39:14.403 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_last_duration": 1, 2026-02-13T02:39:14.403 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_schedule": "queued for scrub", 2026-02-13T02:39:14.403 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_schedule_at": "0", 2026-02-13T02:39:14.403 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_is_future": false, 2026-02-13T02:39:14.403 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_vs_date": false, 2026-02-13T02:39:14.403 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_reported_epoch": 19, 2026-02-13T02:39:14.403 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_seq": 138 2026-02-13T02:39:14.403 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:39:14.404 INFO:tasks.workunit.client.0.trial112.stdout:query output: 2026-02-13T02:39:14.475 INFO:tasks.workunit.client.0.trial112.stdout: "scrubber": { 2026-02-13T02:39:14.475 INFO:tasks.workunit.client.0.trial112.stdout: "active": true, 2026-02-13T02:39:14.475 INFO:tasks.workunit.client.0.trial112.stdout: "epoch_start": "14", 2026-02-13T02:39:14.475 INFO:tasks.workunit.client.0.trial112.stdout: "start": "1:00000000::::0", 2026-02-13T02:39:14.475 INFO:tasks.workunit.client.0.trial112.stdout: "end": "MIN", 2026-02-13T02:39:14.475 INFO:tasks.workunit.client.0.trial112.stdout: "max_end": "MIN", 2026-02-13T02:39:14.475 INFO:tasks.workunit.client.0.trial112.stdout: "subset_last_update": "0'0", 2026-02-13T02:39:14.475 INFO:tasks.workunit.client.0.trial112.stdout: "deep": false, 2026-02-13T02:39:14.475 INFO:tasks.workunit.client.0.trial112.stdout: "req_scrub": false, 2026-02-13T02:39:14.476 INFO:tasks.workunit.client.0.trial112.stdout: "auto_repair": false, 2026-02-13T02:39:14.476 INFO:tasks.workunit.client.0.trial112.stdout: "check_repair": false, 2026-02-13T02:39:14.476 INFO:tasks.workunit.client.0.trial112.stdout: "deep_scrub_on_error": false, 2026-02-13T02:39:14.476 INFO:tasks.workunit.client.0.trial112.stdout: "priority": 5, 2026-02-13T02:39:14.476 INFO:tasks.workunit.client.0.trial112.stdout: "shallow_errors": 0, 2026-02-13T02:39:14.476 INFO:tasks.workunit.client.0.trial112.stdout: "deep_errors": 0, 2026-02-13T02:39:14.476 INFO:tasks.workunit.client.0.trial112.stdout: "fixed": 0, 2026-02-13T02:39:14.476 INFO:tasks.workunit.client.0.trial112.stdout: "waiting_on_whom": [], 2026-02-13T02:39:14.476 INFO:tasks.workunit.client.0.trial112.stdout: "schedule": "scrubbing", 2026-02-13T02:39:14.476 INFO:tasks.workunit.client.0.trial112.stdout: "is_reserving_replicas": false 2026-02-13T02:39:14.476 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:39:14.476 INFO:tasks.workunit.client.0.trial112.stdout: "agent_state": {} 2026-02-13T02:39:14.561 INFO:tasks.workunit.client.0.trial112.stdout:( 2026-02-13T02:39:14.561 INFO:tasks.workunit.client.0.trial112.stdout:[query_epoch]=19 2026-02-13T02:39:14.562 INFO:tasks.workunit.client.0.trial112.stdout:[query_seq]=143 2026-02-13T02:39:14.562 INFO:tasks.workunit.client.0.trial112.stdout:[query_active]=true 2026-02-13T02:39:14.562 INFO:tasks.workunit.client.0.trial112.stdout:[query_schedule]='scrubbing' 2026-02-13T02:39:14.562 INFO:tasks.workunit.client.0.trial112.stdout:[query_schedule_at]='0' 2026-02-13T02:39:14.562 INFO:tasks.workunit.client.0.trial112.stdout:[query_target_at]='0' 2026-02-13T02:39:14.562 INFO:tasks.workunit.client.0.trial112.stdout:[query_last_duration]=1 2026-02-13T02:39:14.562 INFO:tasks.workunit.client.0.trial112.stdout:[query_last_stamp]='2026-02-11T02:39:10.007136+0000' 2026-02-13T02:39:14.562 INFO:tasks.workunit.client.0.trial112.stdout:[query_last_scrub]='17x90' 2026-02-13T02:39:14.562 INFO:tasks.workunit.client.0.trial112.stdout:[query_is_future]=false 2026-02-13T02:39:14.562 INFO:tasks.workunit.client.0.trial112.stdout:[query_vs_date]=false 2026-02-13T02:39:14.562 INFO:tasks.workunit.client.0.trial112.stdout:[query_scrub_seq]=null 2026-02-13T02:39:14.562 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_pg_state]='active+clean' 2026-02-13T02:39:14.563 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_state_has_scrubbing]=false 2026-02-13T02:39:14.563 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_last_duration]=1 2026-02-13T02:39:14.563 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_schedule]='queued for scrub' 2026-02-13T02:39:14.563 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_schedule_at]='0' 2026-02-13T02:39:14.563 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_is_future]=false 2026-02-13T02:39:14.563 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_vs_date]=false 2026-02-13T02:39:14.563 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_reported_epoch]=19 2026-02-13T02:39:14.563 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_seq]=138 2026-02-13T02:39:14.563 INFO:tasks.workunit.client.0.trial112.stdout:) 2026-02-13T02:39:14.572 INFO:tasks.workunit.client.0.trial112.stdout:--> loop: 1 ~ true / 143 / 138 / false / 2026-02-11T02:39:10.007136+0000 / scrubbing %%% dmp_state_has_scrubbing query_active 2026-02-13T02:39:14.572 INFO:tasks.workunit.client.0.trial112.stdout:key is dmp_state_has_scrubbing: negation:0 # expected: true # in actual: false 2026-02-13T02:39:14.572 INFO:tasks.workunit.client.0.trial112.stdout:key is query_active: negation:0 # expected: false # in actual: true 2026-02-13T02:39:15.220 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:39:15.231 INFO:tasks.workunit.client.0.trial112.stdout:from pg dump pg: { 2026-02-13T02:39:15.232 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_pg_state": "active+clean+scrubbing", 2026-02-13T02:39:15.232 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_state_has_scrubbing": true, 2026-02-13T02:39:15.232 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_last_duration": 1, 2026-02-13T02:39:15.232 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_schedule": "scrubbing for 2s", 2026-02-13T02:39:15.232 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_schedule_at": "0", 2026-02-13T02:39:15.232 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_is_future": false, 2026-02-13T02:39:15.232 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_vs_date": false, 2026-02-13T02:39:15.232 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_reported_epoch": 19, 2026-02-13T02:39:15.232 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_seq": 142 2026-02-13T02:39:15.232 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:39:15.233 INFO:tasks.workunit.client.0.trial112.stdout:query output: 2026-02-13T02:39:15.307 INFO:tasks.workunit.client.0.trial112.stdout: "scrubber": { 2026-02-13T02:39:15.308 INFO:tasks.workunit.client.0.trial112.stdout: "active": true, 2026-02-13T02:39:15.308 INFO:tasks.workunit.client.0.trial112.stdout: "epoch_start": "14", 2026-02-13T02:39:15.308 INFO:tasks.workunit.client.0.trial112.stdout: "start": "1:00000000::::0", 2026-02-13T02:39:15.308 INFO:tasks.workunit.client.0.trial112.stdout: "end": "MIN", 2026-02-13T02:39:15.308 INFO:tasks.workunit.client.0.trial112.stdout: "max_end": "MIN", 2026-02-13T02:39:15.308 INFO:tasks.workunit.client.0.trial112.stdout: "subset_last_update": "0'0", 2026-02-13T02:39:15.308 INFO:tasks.workunit.client.0.trial112.stdout: "deep": false, 2026-02-13T02:39:15.308 INFO:tasks.workunit.client.0.trial112.stdout: "req_scrub": false, 2026-02-13T02:39:15.308 INFO:tasks.workunit.client.0.trial112.stdout: "auto_repair": false, 2026-02-13T02:39:15.308 INFO:tasks.workunit.client.0.trial112.stdout: "check_repair": false, 2026-02-13T02:39:15.308 INFO:tasks.workunit.client.0.trial112.stdout: "deep_scrub_on_error": false, 2026-02-13T02:39:15.308 INFO:tasks.workunit.client.0.trial112.stdout: "priority": 5, 2026-02-13T02:39:15.309 INFO:tasks.workunit.client.0.trial112.stdout: "shallow_errors": 0, 2026-02-13T02:39:15.309 INFO:tasks.workunit.client.0.trial112.stdout: "deep_errors": 0, 2026-02-13T02:39:15.309 INFO:tasks.workunit.client.0.trial112.stdout: "fixed": 0, 2026-02-13T02:39:15.309 INFO:tasks.workunit.client.0.trial112.stdout: "waiting_on_whom": [], 2026-02-13T02:39:15.309 INFO:tasks.workunit.client.0.trial112.stdout: "schedule": "scrubbing", 2026-02-13T02:39:15.309 INFO:tasks.workunit.client.0.trial112.stdout: "is_reserving_replicas": false 2026-02-13T02:39:15.309 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:39:15.309 INFO:tasks.workunit.client.0.trial112.stdout: "agent_state": {} 2026-02-13T02:39:15.391 INFO:tasks.workunit.client.0.trial112.stdout:( 2026-02-13T02:39:15.391 INFO:tasks.workunit.client.0.trial112.stdout:[query_epoch]=19 2026-02-13T02:39:15.391 INFO:tasks.workunit.client.0.trial112.stdout:[query_seq]=143 2026-02-13T02:39:15.392 INFO:tasks.workunit.client.0.trial112.stdout:[query_active]=true 2026-02-13T02:39:15.392 INFO:tasks.workunit.client.0.trial112.stdout:[query_schedule]='scrubbing' 2026-02-13T02:39:15.392 INFO:tasks.workunit.client.0.trial112.stdout:[query_schedule_at]='0' 2026-02-13T02:39:15.392 INFO:tasks.workunit.client.0.trial112.stdout:[query_target_at]='0' 2026-02-13T02:39:15.392 INFO:tasks.workunit.client.0.trial112.stdout:[query_last_duration]=1 2026-02-13T02:39:15.392 INFO:tasks.workunit.client.0.trial112.stdout:[query_last_stamp]='2026-02-11T02:39:10.007136+0000' 2026-02-13T02:39:15.392 INFO:tasks.workunit.client.0.trial112.stdout:[query_last_scrub]='17x90' 2026-02-13T02:39:15.392 INFO:tasks.workunit.client.0.trial112.stdout:[query_is_future]=false 2026-02-13T02:39:15.392 INFO:tasks.workunit.client.0.trial112.stdout:[query_vs_date]=false 2026-02-13T02:39:15.392 INFO:tasks.workunit.client.0.trial112.stdout:[query_scrub_seq]=null 2026-02-13T02:39:15.392 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_pg_state]='active+clean+scrubbing' 2026-02-13T02:39:15.392 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_state_has_scrubbing]=true 2026-02-13T02:39:15.393 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_last_duration]=1 2026-02-13T02:39:15.393 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_schedule]='scrubbing for 2s' 2026-02-13T02:39:15.393 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_schedule_at]='0' 2026-02-13T02:39:15.393 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_is_future]=false 2026-02-13T02:39:15.393 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_vs_date]=false 2026-02-13T02:39:15.393 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_reported_epoch]=19 2026-02-13T02:39:15.393 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_seq]=142 2026-02-13T02:39:15.393 INFO:tasks.workunit.client.0.trial112.stdout:) 2026-02-13T02:39:15.403 INFO:tasks.workunit.client.0.trial112.stdout:--> loop: 2 ~ true / 143 / 142 / false / 2026-02-11T02:39:10.007136+0000 / scrubbing %%% dmp_state_has_scrubbing query_active 2026-02-13T02:39:15.404 INFO:tasks.workunit.client.0.trial112.stdout:key is dmp_state_has_scrubbing: negation:0 # expected: true # in actual: true 2026-02-13T02:39:15.404 INFO:tasks.workunit.client.0.trial112.stdout:WaitingActive - 'dmp_state_has_scrubbing' actual value (true) matches expected (true) (negation: 0) 2026-02-13T02:39:15.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:157: wait_any_cond: return 0 2026-02-13T02:39:15.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:648: TEST_dump_scrub_schedule: sleep 4 2026-02-13T02:39:19.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:649: TEST_dump_scrub_schedule: perf_counters td/osd-scrub-test 3 2026-02-13T02:39:19.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2026-02-13T02:39:19.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=3 2026-02-13T02:39:19.406 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 3 - 1 2026-02-13T02:39:19.408 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 2 2026-02-13T02:39:19.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:39:19.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2026-02-13T02:39:19.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-13T02:39:19.485 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:39:19.485 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_ec": [ 2026-02-13T02:39:19.486 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:39:19.486 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:39:19.486 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:39:19.486 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:39:19.486 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:39:19.486 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:39:19.486 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:39:19.486 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:39:19.486 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:39:19.486 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:39:19.486 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:39:19.487 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:39:19.487 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:39:19.487 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:39:19.487 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_repl": [ 2026-02-13T02:39:19.487 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:39:19.487 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:39:19.487 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:39:19.487 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:39:19.487 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:39:19.487 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:39:19.487 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:39:19.487 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:39:19.488 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:39:19.488 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:39:19.488 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:39:19.488 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:39:19.488 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:39:19.488 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:39:19.488 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_ec": [ 2026-02-13T02:39:19.488 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:39:19.488 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:39:19.488 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:39:19.488 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:39:19.488 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:39:19.489 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:39:19.489 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:39:19.489 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:39:19.489 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:39:19.489 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:39:19.489 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:39:19.489 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:39:19.489 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:39:19.489 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:39:19.489 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_repl": [ 2026-02-13T02:39:19.489 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:39:19.489 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:39:19.490 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:39:19.490 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:39:19.490 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:39:19.490 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:39:19.490 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:39:19.490 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:39:19.490 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:39:19.490 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:39:19.490 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:39:19.490 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:39:19.490 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:39:19.490 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:39:19.490 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:39:19.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:39:19.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2026-02-13T02:39:19.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-13T02:39:19.557 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:39:19.557 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_ec": [ 2026-02-13T02:39:19.557 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:39:19.557 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:39:19.557 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:39:19.557 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:39:19.558 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:39:19.558 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:39:19.558 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:39:19.558 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:39:19.558 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:39:19.558 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:39:19.558 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:39:19.558 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:39:19.558 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:39:19.558 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:39:19.558 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_repl": [ 2026-02-13T02:39:19.558 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:39:19.559 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:39:19.559 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:39:19.559 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:39:19.559 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:39:19.559 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:39:19.559 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:39:19.559 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 7, 2026-02-13T02:39:19.559 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:39:19.559 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:39:19.559 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:39:19.559 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:39:19.559 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:39:19.559 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:39:19.560 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_ec": [ 2026-02-13T02:39:19.560 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:39:19.560 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:39:19.560 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:39:19.560 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:39:19.560 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:39:19.560 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:39:19.560 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:39:19.560 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:39:19.560 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:39:19.560 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:39:19.560 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:39:19.560 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:39:19.561 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:39:19.561 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:39:19.561 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_repl": [ 2026-02-13T02:39:19.561 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:39:19.561 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:39:19.561 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:39:19.561 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:39:19.561 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:39:19.561 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:39:19.561 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:39:19.561 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 2, 2026-02-13T02:39:19.561 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:39:19.562 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:39:19.562 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:39:19.562 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:39:19.562 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:39:19.562 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:39:19.562 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:39:19.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:39:19.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.2 counter dump 2026-02-13T02:39:19.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-13T02:39:19.628 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:39:19.629 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_ec": [ 2026-02-13T02:39:19.629 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:39:19.629 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:39:19.629 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:39:19.629 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:39:19.629 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:39:19.629 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:39:19.629 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:39:19.629 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:39:19.629 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:39:19.629 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:39:19.630 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:39:19.630 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:39:19.630 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:39:19.630 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:39:19.630 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_repl": [ 2026-02-13T02:39:19.630 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:39:19.630 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:39:19.630 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:39:19.630 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:39:19.630 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:39:19.630 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:39:19.630 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:39:19.630 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:39:19.631 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:39:19.631 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:39:19.631 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:39:19.631 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:39:19.631 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:39:19.631 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:39:19.631 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_ec": [ 2026-02-13T02:39:19.631 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:39:19.631 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:39:19.631 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:39:19.631 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:39:19.631 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:39:19.631 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:39:19.632 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:39:19.632 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:39:19.632 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:39:19.632 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:39:19.632 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:39:19.632 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:39:19.632 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:39:19.632 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:39:19.632 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_repl": [ 2026-02-13T02:39:19.632 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:39:19.632 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:39:19.632 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:39:19.632 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:39:19.633 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:39:19.633 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:39:19.633 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:39:19.633 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:39:19.633 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:39:19.633 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:39:19.633 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:39:19.633 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:39:19.633 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:39:19.633 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:39:19.633 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:39:19.634 INFO:tasks.workunit.client.0.trial112.stdout:-------------- Teardown Test TEST_dump_scrub_schedule ------------------ 2026-02-13T02:39:19.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_dump_scrub_schedule ------------------' 2026-02-13T02:39:19.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2026-02-13T02:39:19.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-13T02:39:19.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:39:19.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-13T02:39:19.635 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:39:19.635 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:39:19.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:39:19.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:39:19.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:39:19.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:39:19.940 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:39:19.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:39:19.942 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:39:19.944 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:39:19.944 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:39:19.944 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:39:19.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:39:19.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:39:19.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:39:19.947 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:39:19.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:39:19.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:39:19.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-13T02:39:19.961 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:39:19.961 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:39:19.961 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:39:19.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.508946 2026-02-13T02:39:19.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:39:19.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:39:19.963 INFO:tasks.workunit.client.0.trial112.stdout:-------------- Complete Test TEST_dump_scrub_schedule ------------------ 2026-02-13T02:39:19.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_dump_scrub_schedule ------------------' 2026-02-13T02:39:19.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2026-02-13T02:39:19.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_interval_changes -------------------' 2026-02-13T02:39:19.964 INFO:tasks.workunit.client.0.trial112.stdout:-------------- Prepare Test TEST_interval_changes ------------------- 2026-02-13T02:39:19.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2026-02-13T02:39:19.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2026-02-13T02:39:19.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2026-02-13T02:39:19.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-13T02:39:19.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:39:19.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-13T02:39:19.966 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:39:19.966 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:39:19.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:39:19.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:39:19.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:39:19.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:39:19.968 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:39:19.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:39:19.968 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:39:19.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:39:19.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:39:19.970 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:39:19.971 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:39:19.971 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:39:19.971 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:39:19.972 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:39:19.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:39:19.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:39:19.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-13T02:39:19.975 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:39:19.975 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:39:19.975 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:39:19.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.508946 2026-02-13T02:39:19.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:39:19.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:39:19.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2026-02-13T02:39:19.978 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T02:39:19.978 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:39:19.978 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:39:19.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.508946 2026-02-13T02:39:19.979 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T02:39:19.980 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-13T02:39:19.980 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T02:39:19.981 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2026-02-13T02:39:19.981 INFO:tasks.workunit.client.0.trial112.stdout:-------------- Run Test TEST_interval_changes ----------------------- 2026-02-13T02:39:19.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_interval_changes -----------------------' 2026-02-13T02:39:19.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_interval_changes td/osd-scrub-test 2026-02-13T02:39:19.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: TEST_interval_changes: local poolname=test 2026-02-13T02:39:19.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:143: TEST_interval_changes: local OSDS=2 2026-02-13T02:39:19.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: TEST_interval_changes: local objects=10 2026-02-13T02:39:19.982 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:146: TEST_interval_changes: expr 24 '*' 60 '*' 60 2026-02-13T02:39:19.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:146: TEST_interval_changes: local day=86400 2026-02-13T02:39:19.983 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:147: TEST_interval_changes: expr 86400 '*' 7 2026-02-13T02:39:19.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:147: TEST_interval_changes: local week=604800 2026-02-13T02:39:19.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:148: TEST_interval_changes: local min_interval=86400 2026-02-13T02:39:19.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:149: TEST_interval_changes: local max_interval=604800 2026-02-13T02:39:19.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:150: TEST_interval_changes: local WAIT_FOR_UPDATE=15 2026-02-13T02:39:19.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:152: TEST_interval_changes: TESTDATA=testdata.508946 2026-02-13T02:39:19.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:155: TEST_interval_changes: run_mon td/osd-scrub-test a --osd_pool_default_size=2 2026-02-13T02:39:19.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-test 2026-02-13T02:39:19.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-13T02:39:19.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-13T02:39:19.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-13T02:39:19.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-test/a 2026-02-13T02:39:19.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=2 2026-02-13T02:39:20.006 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-13T02:39:20.006 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:39:20.006 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:39:20.006 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:39:20.006 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:39:20.007 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:39:20.007 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:39:20.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2026-02-13T02:39:20.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-13T02:39:20.035 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-13T02:39:20.035 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:39:20.035 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:39:20.035 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-13T02:39:20.036 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-13T02:39:20.037 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:39:20.037 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:39:20.037 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:39:20.037 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:39:20.037 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:39:20.037 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:39:20.037 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.508946/ceph-mon.a.asok 2026-02-13T02:39:20.037 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:39:20.037 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.508946/ceph-mon.a.asok config get fsid 2026-02-13T02:39:20.093 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-13T02:39:20.093 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:39:20.093 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:39:20.093 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-13T02:39:20.093 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-13T02:39:20.094 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:39:20.095 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:39:20.095 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:39:20.095 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:39:20.095 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:39:20.095 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:39:20.095 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.508946/ceph-mon.a.asok 2026-02-13T02:39:20.095 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:39:20.095 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.508946/ceph-mon.a.asok config get mon_host 2026-02-13T02:39:20.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:156: TEST_interval_changes: run_mgr td/osd-scrub-test x --mgr_stats_period=1 2026-02-13T02:39:20.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-test 2026-02-13T02:39:20.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-13T02:39:20.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-13T02:39:20.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-13T02:39:20.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-test/x 2026-02-13T02:39:20.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-13T02:39:20.247 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-13T02:39:20.247 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:39:20.247 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:39:20.247 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:39:20.247 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:39:20.247 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:39:20.248 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:39:20.248 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:39:20.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mgr_stats_period=1 2026-02-13T02:39:20.267 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:157: TEST_interval_changes: expr 2 - 1 2026-02-13T02:39:20.268 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:157: TEST_interval_changes: seq 0 1 2026-02-13T02:39:20.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:157: TEST_interval_changes: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:39:20.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:159: TEST_interval_changes: run_osd td/osd-scrub-test 0 --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 2026-02-13T02:39:20.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-13T02:39:20.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:39:20.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-13T02:39:20.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:39:20.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/0 2026-02-13T02:39:20.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-13T02:39:20.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:39:20.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:39:20.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:39:20.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2026-02-13T02:39:20.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2026-02-13T02:39:20.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:39:20.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:39:20.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-13T02:39:20.273 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:39:20.273 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:39:20.274 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:39:20.274 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:39:20.274 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:39:20.274 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:39:20.274 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:39:20.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:39:20.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:39:20.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:39:20.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:39:20.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-13T02:39:20.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-13T02:39:20.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:39:20.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:39:20.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:39:20.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:39:20.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:39:20.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0' 2026-02-13T02:39:20.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/0 2026-02-13T02:39:20.275 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:39:20.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e1d6e51a-5f20-4658-8376-eae55535b83e 2026-02-13T02:39:20.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 e1d6e51a-5f20-4658-8376-eae55535b83e' 2026-02-13T02:39:20.276 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 e1d6e51a-5f20-4658-8376-eae55535b83e 2026-02-13T02:39:20.276 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:39:20.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDYjo5pz2GWEBAAgN81c2AHa0Fe6sWK9CfzLw== 2026-02-13T02:39:20.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDYjo5pz2GWEBAAgN81c2AHa0Fe6sWK9CfzLw=="}' 2026-02-13T02:39:20.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e1d6e51a-5f20-4658-8376-eae55535b83e -i td/osd-scrub-test/0/new.json 2026-02-13T02:39:20.372 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:39:20.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/0/new.json 2026-02-13T02:39:20.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 --mkfs --key AQDYjo5pz2GWEBAAgN81c2AHa0Fe6sWK9CfzLw== --osd-uuid e1d6e51a-5f20-4658-8376-eae55535b83e 2026-02-13T02:39:20.396 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:39:20.387+0000 7f07b1105a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:39:20.398 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:39:20.387+0000 7f07b1105a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:39:20.399 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:39:20.387+0000 7f07b1105a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:39:20.399 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:39:20.391+0000 7f07b1105a40 -1 bdev(0x55daf577ac00 td/osd-scrub-test/0/block) open stat got: (1) Operation not permitted 2026-02-13T02:39:20.399 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:39:20.391+0000 7f07b1105a40 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2026-02-13T02:39:21.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2026-02-13T02:39:21.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:39:21.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-13T02:39:21.140 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T02:39:21.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:39:21.243 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T02:39:21.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-13T02:39:21.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 2026-02-13T02:39:21.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:39:21.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:39:21.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:39:21.266 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:39:21.255+0000 7f160fa3ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:39:21.268 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:39:21.259+0000 7f160fa3ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:39:21.269 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:39:21.259+0000 7f160fa3ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:39:21.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-13T02:39:21.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:39:21.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:39:21.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:39:21.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:39:21.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:39:21.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:39:21.341 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:39:21.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:39:21.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:39:21.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:39:21.769 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:39:21.759+0000 7f160fa3ca40 -1 Falling back to public interface 2026-02-13T02:39:22.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:39:22.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:39:22.433 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:39:22.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:39:22.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:39:22.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:39:22.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:39:22.741 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:39:22.731+0000 7f160fa3ca40 -1 osd.0 0 log_to_monitors true 2026-02-13T02:39:23.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:39:23.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:39:23.526 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:39:23.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:39:23.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:39:23.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:39:23.683 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3503029056,v1:127.0.0.1:6801/3503029056] [v2:127.0.0.1:6802/3503029056,v1:127.0.0.1:6803/3503029056] exists,up e1d6e51a-5f20-4658-8376-eae55535b83e 2026-02-13T02:39:23.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:39:23.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:39:23.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:39:23.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:157: TEST_interval_changes: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:39:23.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:159: TEST_interval_changes: run_osd td/osd-scrub-test 1 --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 2026-02-13T02:39:23.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-13T02:39:23.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:39:23.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-13T02:39:23.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:39:23.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/1 2026-02-13T02:39:23.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-13T02:39:23.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:39:23.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:39:23.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:39:23.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2026-02-13T02:39:23.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2026-02-13T02:39:23.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:39:23.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:39:23.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-13T02:39:23.688 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:39:23.689 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:39:23.689 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:39:23.689 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:39:23.689 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:39:23.689 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:39:23.689 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:39:23.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:39:23.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:39:23.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:39:23.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:39:23.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-13T02:39:23.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-13T02:39:23.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:39:23.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:39:23.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:39:23.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:39:23.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:39:23.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0' 2026-02-13T02:39:23.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/1 2026-02-13T02:39:23.690 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:39:23.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=344f4721-6a82-4f70-9ca2-4638d9b84017 2026-02-13T02:39:23.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 344f4721-6a82-4f70-9ca2-4638d9b84017' 2026-02-13T02:39:23.691 INFO:tasks.workunit.client.0.trial112.stdout:add osd1 344f4721-6a82-4f70-9ca2-4638d9b84017 2026-02-13T02:39:23.691 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:39:23.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDbjo5pKlV3KRAAeDrBQGHhjasSfI9JH9roeg== 2026-02-13T02:39:23.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDbjo5pKlV3KRAAeDrBQGHhjasSfI9JH9roeg=="}' 2026-02-13T02:39:23.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 344f4721-6a82-4f70-9ca2-4638d9b84017 -i td/osd-scrub-test/1/new.json 2026-02-13T02:39:23.861 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:39:23.871 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/1/new.json 2026-02-13T02:39:23.872 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 --mkfs --key AQDbjo5pKlV3KRAAeDrBQGHhjasSfI9JH9roeg== --osd-uuid 344f4721-6a82-4f70-9ca2-4638d9b84017 2026-02-13T02:39:23.894 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:39:23.883+0000 7fc4e488aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:39:23.896 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:39:23.887+0000 7fc4e488aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:39:23.897 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:39:23.887+0000 7fc4e488aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:39:23.898 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:39:23.887+0000 7fc4e488aa40 -1 bdev(0x55c672e2cc00 td/osd-scrub-test/1/block) open stat got: (1) Operation not permitted 2026-02-13T02:39:23.898 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:39:23.887+0000 7fc4e488aa40 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2026-02-13T02:39:24.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2026-02-13T02:39:24.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:39:24.492 INFO:tasks.workunit.client.0.trial112.stdout:adding osd1 key to auth repository 2026-02-13T02:39:24.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-13T02:39:24.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:39:24.691 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T02:39:24.692 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-13T02:39:24.692 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 2026-02-13T02:39:24.692 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:39:24.692 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:39:24.692 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:39:24.706 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:39:24.695+0000 7f590cfeda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:39:24.708 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:39:24.699+0000 7f590cfeda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:39:24.708 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:39:24.699+0000 7f590cfeda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:39:24.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-13T02:39:24.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:39:24.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:39:24.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:39:24.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:39:24.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:39:24.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:39:24.856 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:39:24.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:39:24.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:39:25.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:39:25.092 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:39:25.083+0000 7f590cfeda40 -1 Falling back to public interface 2026-02-13T02:39:25.641 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:39:25.631+0000 7f590cfeda40 -1 osd.1 0 log_to_monitors true 2026-02-13T02:39:26.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:39:26.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:39:26.011 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:39:26.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:39:26.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:39:26.012 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:39:26.170 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:39:27.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:39:27.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:39:27.172 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:39:27.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:39:27.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:39:27.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:39:27.331 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1909109138,v1:127.0.0.1:6811/1909109138] [v2:127.0.0.1:6812/1909109138,v1:127.0.0.1:6813/1909109138] exists,up 344f4721-6a82-4f70-9ca2-4638d9b84017 2026-02-13T02:39:27.332 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:39:27.332 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:39:27.332 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:39:27.332 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:163: TEST_interval_changes: create_pool test 1 1 2026-02-13T02:39:27.332 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2026-02-13T02:39:27.521 INFO:tasks.workunit.client.0.trial112.stderr:pool 'test' created 2026-02-13T02:39:27.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:39:28.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:164: TEST_interval_changes: wait_for_clean 2026-02-13T02:39:28.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:39:28.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:39:28.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:39:28.534 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:39:28.534 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:39:28.535 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:39:28.535 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:39:28.535 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:39:28.535 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:39:28.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:39:28.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:39:28.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:39:28.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:39:28.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:39:28.647 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:39:28.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:39:28.812 INFO:tasks.workunit.client.0.trial112.stderr:1' 2026-02-13T02:39:28.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:39:28.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:39:28.812 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:39:28.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836486 2026-02-13T02:39:28.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836486 2026-02-13T02:39:28.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486' 2026-02-13T02:39:28.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:39:28.884 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:39:28.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705668 2026-02-13T02:39:28.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705668 2026-02-13T02:39:28.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486 1-38654705668' 2026-02-13T02:39:28.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:39:28.958 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836486 2026-02-13T02:39:28.959 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:39:28.959 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:39:28.960 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836486 2026-02-13T02:39:28.960 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:39:28.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836486 2026-02-13T02:39:28.961 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836486 2026-02-13T02:39:28.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2026-02-13T02:39:28.962 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:39:29.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 21474836486 2026-02-13T02:39:29.132 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:39:30.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:39:30.133 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:39:30.289 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 21474836486 2026-02-13T02:39:30.289 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:39:31.290 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 499 -eq 0 ']' 2026-02-13T02:39:31.291 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:39:31.456 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836486 2026-02-13T02:39:31.456 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:39:31.457 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705668 2026-02-13T02:39:31.457 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:39:31.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:39:31.458 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705668 2026-02-13T02:39:31.459 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:39:31.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705668 2026-02-13T02:39:31.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705668' 2026-02-13T02:39:31.460 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705668 2026-02-13T02:39:31.461 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:39:31.620 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705669 -lt 38654705668 2026-02-13T02:39:31.620 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:39:31.621 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:39:31.621 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:39:31.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:39:31.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:39:31.812 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:39:31.812 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:39:31.812 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:39:31.812 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:39:31.812 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:39:31.812 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:39:31.970 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:39:31.970 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:39:31.971 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:39:31.971 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:39:32.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:39:32.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:39:32.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:39:32.162 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:165: TEST_interval_changes: ceph osd dump 2026-02-13T02:39:32.162 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:165: TEST_interval_changes: grep '^pool.*['\'']test['\'']' 2026-02-13T02:39:32.163 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:165: TEST_interval_changes: awk '{ print $2 }' 2026-02-13T02:39:32.319 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:165: TEST_interval_changes: local poolid=1 2026-02-13T02:39:32.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:167: TEST_interval_changes: dd if=/dev/urandom of=testdata.508946 bs=1032 count=1 2026-02-13T02:39:32.320 INFO:tasks.workunit.client.0.trial112.stderr:1+0 records in 2026-02-13T02:39:32.320 INFO:tasks.workunit.client.0.trial112.stderr:1+0 records out 2026-02-13T02:39:32.320 INFO:tasks.workunit.client.0.trial112.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 3.5787e-05 s, 28.8 MB/s 2026-02-13T02:39:32.321 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:168: TEST_interval_changes: seq 1 10 2026-02-13T02:39:32.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:168: TEST_interval_changes: for i in `seq 1 $objects` 2026-02-13T02:39:32.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:170: TEST_interval_changes: rados -p test put obj1 testdata.508946 2026-02-13T02:39:32.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:168: TEST_interval_changes: for i in `seq 1 $objects` 2026-02-13T02:39:32.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:170: TEST_interval_changes: rados -p test put obj2 testdata.508946 2026-02-13T02:39:32.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:168: TEST_interval_changes: for i in `seq 1 $objects` 2026-02-13T02:39:32.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:170: TEST_interval_changes: rados -p test put obj3 testdata.508946 2026-02-13T02:39:32.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:168: TEST_interval_changes: for i in `seq 1 $objects` 2026-02-13T02:39:32.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:170: TEST_interval_changes: rados -p test put obj4 testdata.508946 2026-02-13T02:39:32.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:168: TEST_interval_changes: for i in `seq 1 $objects` 2026-02-13T02:39:32.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:170: TEST_interval_changes: rados -p test put obj5 testdata.508946 2026-02-13T02:39:32.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:168: TEST_interval_changes: for i in `seq 1 $objects` 2026-02-13T02:39:32.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:170: TEST_interval_changes: rados -p test put obj6 testdata.508946 2026-02-13T02:39:32.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:168: TEST_interval_changes: for i in `seq 1 $objects` 2026-02-13T02:39:32.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:170: TEST_interval_changes: rados -p test put obj7 testdata.508946 2026-02-13T02:39:32.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:168: TEST_interval_changes: for i in `seq 1 $objects` 2026-02-13T02:39:32.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:170: TEST_interval_changes: rados -p test put obj8 testdata.508946 2026-02-13T02:39:32.511 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:168: TEST_interval_changes: for i in `seq 1 $objects` 2026-02-13T02:39:32.512 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:170: TEST_interval_changes: rados -p test put obj9 testdata.508946 2026-02-13T02:39:32.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:168: TEST_interval_changes: for i in `seq 1 $objects` 2026-02-13T02:39:32.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:170: TEST_interval_changes: rados -p test put obj10 testdata.508946 2026-02-13T02:39:32.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:172: TEST_interval_changes: rm -f testdata.508946 2026-02-13T02:39:32.553 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:174: TEST_interval_changes: get_primary test obj1 2026-02-13T02:39:32.553 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2026-02-13T02:39:32.553 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-13T02:39:32.553 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2026-02-13T02:39:32.554 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T02:39:32.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:174: TEST_interval_changes: local primary=1 2026-02-13T02:39:32.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: check_dump_scrubs 1 '1 day' 2026-02-13T02:39:32.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:132: check_dump_scrubs: local primary=1 2026-02-13T02:39:32.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:133: check_dump_scrubs: local 'sched_time_check=1 day' 2026-02-13T02:39:32.716 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: get_asok_path osd.1 2026-02-13T02:39:32.716 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-13T02:39:32.716 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-13T02:39:32.716 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:39:32.716 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:39:32.716 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:39:32.716 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.508946/ceph-osd.1.asok 2026-02-13T02:39:32.717 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: CEPH_ARGS= 2026-02-13T02:39:32.717 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: ceph --admin-daemon /tmp/ceph-asok.508946/ceph-osd.1.asok dump_scrubs 2026-02-13T02:39:32.775 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: DS='[ 2026-02-13T02:39:32.775 INFO:tasks.workunit.client.0.trial112.stderr: { 2026-02-13T02:39:32.775 INFO:tasks.workunit.client.0.trial112.stderr: "pgid": "1.0", 2026-02-13T02:39:32.775 INFO:tasks.workunit.client.0.trial112.stderr: "sched_time": "2026-02-14T02:39:27.514665+0000", 2026-02-13T02:39:32.775 INFO:tasks.workunit.client.0.trial112.stderr: "orig_sched_time": "2026-02-14T02:39:27.514665+0000", 2026-02-13T02:39:32.775 INFO:tasks.workunit.client.0.trial112.stderr: "forced": false, 2026-02-13T02:39:32.776 INFO:tasks.workunit.client.0.trial112.stderr: "level": "shallow", 2026-02-13T02:39:32.776 INFO:tasks.workunit.client.0.trial112.stderr: "urgency": "periodic-regular", 2026-02-13T02:39:32.776 INFO:tasks.workunit.client.0.trial112.stderr: "eligible": false, 2026-02-13T02:39:32.776 INFO:tasks.workunit.client.0.trial112.stderr: "last_issue": "ok" 2026-02-13T02:39:32.776 INFO:tasks.workunit.client.0.trial112.stderr: }, 2026-02-13T02:39:32.776 INFO:tasks.workunit.client.0.trial112.stderr: { 2026-02-13T02:39:32.776 INFO:tasks.workunit.client.0.trial112.stderr: "pgid": "1.0", 2026-02-13T02:39:32.776 INFO:tasks.workunit.client.0.trial112.stderr: "sched_time": "2026-02-17T21:10:33.728759+0000", 2026-02-13T02:39:32.776 INFO:tasks.workunit.client.0.trial112.stderr: "orig_sched_time": "2026-02-17T21:10:33.728759+0000", 2026-02-13T02:39:32.776 INFO:tasks.workunit.client.0.trial112.stderr: "forced": false, 2026-02-13T02:39:32.776 INFO:tasks.workunit.client.0.trial112.stderr: "level": "deep", 2026-02-13T02:39:32.777 INFO:tasks.workunit.client.0.trial112.stderr: "urgency": "periodic-regular", 2026-02-13T02:39:32.777 INFO:tasks.workunit.client.0.trial112.stderr: "eligible": false, 2026-02-13T02:39:32.777 INFO:tasks.workunit.client.0.trial112.stderr: "last_issue": "ok" 2026-02-13T02:39:32.777 INFO:tasks.workunit.client.0.trial112.stderr: } 2026-02-13T02:39:32.777 INFO:tasks.workunit.client.0.trial112.stderr:]' 2026-02-13T02:39:32.777 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: jq '.[0].sched_time' 2026-02-13T02:39:32.778 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2026-02-14T02:39:27.514665+0000",' '"orig_sched_time":' '"2026-02-14T02:39:27.514665+0000",' '"forced":' false, '"level":' '"shallow",' '"urgency":' '"periodic-regular",' '"eligible":' false, '"last_issue":' '"ok"' '},' '{' '"pgid":' '"1.0",' '"sched_time":' '"2026-02-17T21:10:33.728759+0000",' '"orig_sched_time":' '"2026-02-17T21:10:33.728759+0000",' '"forced":' false, '"level":' '"deep",' '"urgency":' '"periodic-regular",' '"eligible":' false, '"last_issue":' '"ok"' '}' ']' 2026-02-13T02:39:32.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: eval 'SCHED_TIME="2026-02-14T02:39:27.514665+0000"' 2026-02-13T02:39:32.787 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: SCHED_TIME=2026-02-14T02:39:27.514665+0000 2026-02-13T02:39:32.788 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: echo 2026-02-14T02:39:27.514665+0000 2026-02-13T02:39:32.788 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2026-02-13T02:39:32.789 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: date +%Y-%m-%d -d 'now + 1 day' 2026-02-13T02:39:32.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: test 2026-02-14 = 2026-02-14 2026-02-13T02:39:32.791 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:180: TEST_interval_changes: get_asok_path osd.1 2026-02-13T02:39:32.791 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-13T02:39:32.791 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-13T02:39:32.792 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:39:32.792 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:39:32.792 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:39:32.792 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.508946/ceph-osd.1.asok 2026-02-13T02:39:32.793 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:180: TEST_interval_changes: expr 86400 '*' 2 2026-02-13T02:39:32.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:180: TEST_interval_changes: CEPH_ARGS= 2026-02-13T02:39:32.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:180: TEST_interval_changes: ceph --admin-daemon /tmp/ceph-asok.508946/ceph-osd.1.asok config set osd_scrub_min_interval 172800 2026-02-13T02:39:32.849 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:39:32.849 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_min_interval = '' " 2026-02-13T02:39:32.849 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:39:32.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:181: TEST_interval_changes: sleep 15 2026-02-13T02:39:47.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:182: TEST_interval_changes: check_dump_scrubs 1 '2 days' 2026-02-13T02:39:47.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:132: check_dump_scrubs: local primary=1 2026-02-13T02:39:47.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:133: check_dump_scrubs: local 'sched_time_check=2 days' 2026-02-13T02:39:47.857 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: get_asok_path osd.1 2026-02-13T02:39:47.857 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-13T02:39:47.857 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-13T02:39:47.858 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:39:47.858 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:39:47.858 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:39:47.858 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.508946/ceph-osd.1.asok 2026-02-13T02:39:47.859 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: CEPH_ARGS= 2026-02-13T02:39:47.859 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: ceph --admin-daemon /tmp/ceph-asok.508946/ceph-osd.1.asok dump_scrubs 2026-02-13T02:39:47.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: DS='[ 2026-02-13T02:39:47.918 INFO:tasks.workunit.client.0.trial112.stderr: { 2026-02-13T02:39:47.918 INFO:tasks.workunit.client.0.trial112.stderr: "pgid": "1.0", 2026-02-13T02:39:47.918 INFO:tasks.workunit.client.0.trial112.stderr: "sched_time": "2026-02-15T02:39:27.514665+0000", 2026-02-13T02:39:47.918 INFO:tasks.workunit.client.0.trial112.stderr: "orig_sched_time": "2026-02-15T02:39:27.514665+0000", 2026-02-13T02:39:47.918 INFO:tasks.workunit.client.0.trial112.stderr: "forced": false, 2026-02-13T02:39:47.918 INFO:tasks.workunit.client.0.trial112.stderr: "level": "shallow", 2026-02-13T02:39:47.919 INFO:tasks.workunit.client.0.trial112.stderr: "urgency": "periodic-regular", 2026-02-13T02:39:47.919 INFO:tasks.workunit.client.0.trial112.stderr: "eligible": false, 2026-02-13T02:39:47.919 INFO:tasks.workunit.client.0.trial112.stderr: "last_issue": "ok" 2026-02-13T02:39:47.919 INFO:tasks.workunit.client.0.trial112.stderr: }, 2026-02-13T02:39:47.919 INFO:tasks.workunit.client.0.trial112.stderr: { 2026-02-13T02:39:47.919 INFO:tasks.workunit.client.0.trial112.stderr: "pgid": "1.0", 2026-02-13T02:39:47.919 INFO:tasks.workunit.client.0.trial112.stderr: "sched_time": "2026-02-21T04:12:46.050958+0000", 2026-02-13T02:39:47.919 INFO:tasks.workunit.client.0.trial112.stderr: "orig_sched_time": "2026-02-21T04:12:46.050958+0000", 2026-02-13T02:39:47.919 INFO:tasks.workunit.client.0.trial112.stderr: "forced": false, 2026-02-13T02:39:47.919 INFO:tasks.workunit.client.0.trial112.stderr: "level": "deep", 2026-02-13T02:39:47.919 INFO:tasks.workunit.client.0.trial112.stderr: "urgency": "periodic-regular", 2026-02-13T02:39:47.919 INFO:tasks.workunit.client.0.trial112.stderr: "eligible": false, 2026-02-13T02:39:47.919 INFO:tasks.workunit.client.0.trial112.stderr: "last_issue": "ok" 2026-02-13T02:39:47.919 INFO:tasks.workunit.client.0.trial112.stderr: } 2026-02-13T02:39:47.919 INFO:tasks.workunit.client.0.trial112.stderr:]' 2026-02-13T02:39:47.919 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2026-02-15T02:39:27.514665+0000",' '"orig_sched_time":' '"2026-02-15T02:39:27.514665+0000",' '"forced":' false, '"level":' '"shallow",' '"urgency":' '"periodic-regular",' '"eligible":' false, '"last_issue":' '"ok"' '},' '{' '"pgid":' '"1.0",' '"sched_time":' '"2026-02-21T04:12:46.050958+0000",' '"orig_sched_time":' '"2026-02-21T04:12:46.050958+0000",' '"forced":' false, '"level":' '"deep",' '"urgency":' '"periodic-regular",' '"eligible":' false, '"last_issue":' '"ok"' '}' ']' 2026-02-13T02:39:47.920 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: jq '.[0].sched_time' 2026-02-13T02:39:47.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: eval 'SCHED_TIME="2026-02-15T02:39:27.514665+0000"' 2026-02-13T02:39:47.930 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: SCHED_TIME=2026-02-15T02:39:27.514665+0000 2026-02-13T02:39:47.931 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: echo 2026-02-15T02:39:27.514665+0000 2026-02-13T02:39:47.931 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2026-02-13T02:39:47.933 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: date +%Y-%m-%d -d 'now + 2 days' 2026-02-13T02:39:47.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: test 2026-02-15 = 2026-02-15 2026-02-13T02:39:47.935 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:185: TEST_interval_changes: get_asok_path osd.1 2026-02-13T02:39:47.935 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-13T02:39:47.935 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-13T02:39:47.935 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:39:47.935 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:39:47.935 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:39:47.935 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.508946/ceph-osd.1.asok 2026-02-13T02:39:47.936 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:185: TEST_interval_changes: expr 604800 '*' 2 2026-02-13T02:39:47.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:185: TEST_interval_changes: CEPH_ARGS= 2026-02-13T02:39:47.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:185: TEST_interval_changes: ceph --admin-daemon /tmp/ceph-asok.508946/ceph-osd.1.asok config set osd_scrub_max_interval 1209600 2026-02-13T02:39:47.991 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:39:47.991 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_max_interval = '' " 2026-02-13T02:39:47.992 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:39:47.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:186: TEST_interval_changes: sleep 15 2026-02-13T02:40:02.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:187: TEST_interval_changes: check_dump_scrubs 1 '2 days' 2026-02-13T02:40:02.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:132: check_dump_scrubs: local primary=1 2026-02-13T02:40:02.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:133: check_dump_scrubs: local 'sched_time_check=2 days' 2026-02-13T02:40:03.000 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: get_asok_path osd.1 2026-02-13T02:40:03.000 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-13T02:40:03.000 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-13T02:40:03.001 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:40:03.002 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:40:03.002 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:40:03.002 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.508946/ceph-osd.1.asok 2026-02-13T02:40:03.002 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: CEPH_ARGS= 2026-02-13T02:40:03.002 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: ceph --admin-daemon /tmp/ceph-asok.508946/ceph-osd.1.asok dump_scrubs 2026-02-13T02:40:03.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: DS='[ 2026-02-13T02:40:03.060 INFO:tasks.workunit.client.0.trial112.stderr: { 2026-02-13T02:40:03.060 INFO:tasks.workunit.client.0.trial112.stderr: "pgid": "1.0", 2026-02-13T02:40:03.060 INFO:tasks.workunit.client.0.trial112.stderr: "sched_time": "2026-02-15T02:39:27.514665+0000", 2026-02-13T02:40:03.060 INFO:tasks.workunit.client.0.trial112.stderr: "orig_sched_time": "2026-02-15T02:39:27.514665+0000", 2026-02-13T02:40:03.061 INFO:tasks.workunit.client.0.trial112.stderr: "forced": false, 2026-02-13T02:40:03.061 INFO:tasks.workunit.client.0.trial112.stderr: "level": "shallow", 2026-02-13T02:40:03.061 INFO:tasks.workunit.client.0.trial112.stderr: "urgency": "periodic-regular", 2026-02-13T02:40:03.061 INFO:tasks.workunit.client.0.trial112.stderr: "eligible": false, 2026-02-13T02:40:03.061 INFO:tasks.workunit.client.0.trial112.stderr: "last_issue": "ok" 2026-02-13T02:40:03.061 INFO:tasks.workunit.client.0.trial112.stderr: }, 2026-02-13T02:40:03.061 INFO:tasks.workunit.client.0.trial112.stderr: { 2026-02-13T02:40:03.061 INFO:tasks.workunit.client.0.trial112.stderr: "pgid": "1.0", 2026-02-13T02:40:03.061 INFO:tasks.workunit.client.0.trial112.stderr: "sched_time": "2026-02-19T15:56:19.643705+0000", 2026-02-13T02:40:03.061 INFO:tasks.workunit.client.0.trial112.stderr: "orig_sched_time": "2026-02-19T15:56:19.643705+0000", 2026-02-13T02:40:03.061 INFO:tasks.workunit.client.0.trial112.stderr: "forced": false, 2026-02-13T02:40:03.062 INFO:tasks.workunit.client.0.trial112.stderr: "level": "deep", 2026-02-13T02:40:03.062 INFO:tasks.workunit.client.0.trial112.stderr: "urgency": "periodic-regular", 2026-02-13T02:40:03.062 INFO:tasks.workunit.client.0.trial112.stderr: "eligible": false, 2026-02-13T02:40:03.062 INFO:tasks.workunit.client.0.trial112.stderr: "last_issue": "ok" 2026-02-13T02:40:03.062 INFO:tasks.workunit.client.0.trial112.stderr: } 2026-02-13T02:40:03.062 INFO:tasks.workunit.client.0.trial112.stderr:]' 2026-02-13T02:40:03.063 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: jq '.[0].sched_time' 2026-02-13T02:40:03.063 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2026-02-15T02:39:27.514665+0000",' '"orig_sched_time":' '"2026-02-15T02:39:27.514665+0000",' '"forced":' false, '"level":' '"shallow",' '"urgency":' '"periodic-regular",' '"eligible":' false, '"last_issue":' '"ok"' '},' '{' '"pgid":' '"1.0",' '"sched_time":' '"2026-02-19T15:56:19.643705+0000",' '"orig_sched_time":' '"2026-02-19T15:56:19.643705+0000",' '"forced":' false, '"level":' '"deep",' '"urgency":' '"periodic-regular",' '"eligible":' false, '"last_issue":' '"ok"' '}' ']' 2026-02-13T02:40:03.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: eval 'SCHED_TIME="2026-02-15T02:39:27.514665+0000"' 2026-02-13T02:40:03.071 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: SCHED_TIME=2026-02-15T02:39:27.514665+0000 2026-02-13T02:40:03.072 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: echo 2026-02-15T02:39:27.514665+0000 2026-02-13T02:40:03.072 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2026-02-13T02:40:03.074 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: date +%Y-%m-%d -d 'now + 2 days' 2026-02-13T02:40:03.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: test 2026-02-15 = 2026-02-15 2026-02-13T02:40:03.075 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:190: TEST_interval_changes: expr 86400 '*' 3 2026-02-13T02:40:03.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:190: TEST_interval_changes: ceph osd pool set test scrub_min_interval 259200 2026-02-13T02:40:03.280 INFO:tasks.workunit.client.0.trial112.stderr:set pool 1 scrub_min_interval to 259200 2026-02-13T02:40:03.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:191: TEST_interval_changes: sleep 15 2026-02-13T02:40:18.295 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:192: TEST_interval_changes: check_dump_scrubs 1 '3 days' 2026-02-13T02:40:18.295 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:132: check_dump_scrubs: local primary=1 2026-02-13T02:40:18.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:133: check_dump_scrubs: local 'sched_time_check=3 days' 2026-02-13T02:40:18.296 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: get_asok_path osd.1 2026-02-13T02:40:18.297 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-13T02:40:18.297 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-13T02:40:18.297 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:40:18.297 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:40:18.297 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:40:18.298 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.508946/ceph-osd.1.asok 2026-02-13T02:40:18.298 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: CEPH_ARGS= 2026-02-13T02:40:18.298 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: ceph --admin-daemon /tmp/ceph-asok.508946/ceph-osd.1.asok dump_scrubs 2026-02-13T02:40:18.358 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: DS='[ 2026-02-13T02:40:18.358 INFO:tasks.workunit.client.0.trial112.stderr: { 2026-02-13T02:40:18.358 INFO:tasks.workunit.client.0.trial112.stderr: "pgid": "1.0", 2026-02-13T02:40:18.358 INFO:tasks.workunit.client.0.trial112.stderr: "sched_time": "2026-02-16T02:39:27.514665+0000", 2026-02-13T02:40:18.358 INFO:tasks.workunit.client.0.trial112.stderr: "orig_sched_time": "2026-02-16T02:39:27.514665+0000", 2026-02-13T02:40:18.358 INFO:tasks.workunit.client.0.trial112.stderr: "forced": false, 2026-02-13T02:40:18.358 INFO:tasks.workunit.client.0.trial112.stderr: "level": "shallow", 2026-02-13T02:40:18.358 INFO:tasks.workunit.client.0.trial112.stderr: "urgency": "periodic-regular", 2026-02-13T02:40:18.358 INFO:tasks.workunit.client.0.trial112.stderr: "eligible": false, 2026-02-13T02:40:18.358 INFO:tasks.workunit.client.0.trial112.stderr: "last_issue": "ok" 2026-02-13T02:40:18.358 INFO:tasks.workunit.client.0.trial112.stderr: }, 2026-02-13T02:40:18.358 INFO:tasks.workunit.client.0.trial112.stderr: { 2026-02-13T02:40:18.359 INFO:tasks.workunit.client.0.trial112.stderr: "pgid": "1.0", 2026-02-13T02:40:18.359 INFO:tasks.workunit.client.0.trial112.stderr: "sched_time": "2026-02-18T03:46:14.366082+0000", 2026-02-13T02:40:18.359 INFO:tasks.workunit.client.0.trial112.stderr: "orig_sched_time": "2026-02-18T03:46:14.366082+0000", 2026-02-13T02:40:18.359 INFO:tasks.workunit.client.0.trial112.stderr: "forced": false, 2026-02-13T02:40:18.359 INFO:tasks.workunit.client.0.trial112.stderr: "level": "deep", 2026-02-13T02:40:18.359 INFO:tasks.workunit.client.0.trial112.stderr: "urgency": "periodic-regular", 2026-02-13T02:40:18.359 INFO:tasks.workunit.client.0.trial112.stderr: "eligible": false, 2026-02-13T02:40:18.359 INFO:tasks.workunit.client.0.trial112.stderr: "last_issue": "ok" 2026-02-13T02:40:18.359 INFO:tasks.workunit.client.0.trial112.stderr: } 2026-02-13T02:40:18.359 INFO:tasks.workunit.client.0.trial112.stderr:]' 2026-02-13T02:40:18.360 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: jq '.[0].sched_time' 2026-02-13T02:40:18.360 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2026-02-16T02:39:27.514665+0000",' '"orig_sched_time":' '"2026-02-16T02:39:27.514665+0000",' '"forced":' false, '"level":' '"shallow",' '"urgency":' '"periodic-regular",' '"eligible":' false, '"last_issue":' '"ok"' '},' '{' '"pgid":' '"1.0",' '"sched_time":' '"2026-02-18T03:46:14.366082+0000",' '"orig_sched_time":' '"2026-02-18T03:46:14.366082+0000",' '"forced":' false, '"level":' '"deep",' '"urgency":' '"periodic-regular",' '"eligible":' false, '"last_issue":' '"ok"' '}' ']' 2026-02-13T02:40:18.368 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: eval 'SCHED_TIME="2026-02-16T02:39:27.514665+0000"' 2026-02-13T02:40:18.368 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: SCHED_TIME=2026-02-16T02:39:27.514665+0000 2026-02-13T02:40:18.369 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: echo 2026-02-16T02:39:27.514665+0000 2026-02-13T02:40:18.369 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2026-02-13T02:40:18.371 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: date +%Y-%m-%d -d 'now + 3 days' 2026-02-13T02:40:18.372 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: test 2026-02-16 = 2026-02-16 2026-02-13T02:40:18.372 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:195: TEST_interval_changes: expr 604800 '*' 3 2026-02-13T02:40:18.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:195: TEST_interval_changes: ceph osd pool set test scrub_max_interval 1814400 2026-02-13T02:40:18.573 INFO:tasks.workunit.client.0.trial112.stderr:set pool 1 scrub_max_interval to 1814400 2026-02-13T02:40:18.585 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:196: TEST_interval_changes: sleep 15 2026-02-13T02:40:33.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:197: TEST_interval_changes: check_dump_scrubs 1 '3 days' 2026-02-13T02:40:33.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:132: check_dump_scrubs: local primary=1 2026-02-13T02:40:33.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:133: check_dump_scrubs: local 'sched_time_check=3 days' 2026-02-13T02:40:33.587 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: get_asok_path osd.1 2026-02-13T02:40:33.588 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-13T02:40:33.588 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-13T02:40:33.589 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:40:33.589 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:40:33.589 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:40:33.589 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.508946/ceph-osd.1.asok 2026-02-13T02:40:33.589 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: CEPH_ARGS= 2026-02-13T02:40:33.590 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: ceph --admin-daemon /tmp/ceph-asok.508946/ceph-osd.1.asok dump_scrubs 2026-02-13T02:40:33.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: DS='[ 2026-02-13T02:40:33.648 INFO:tasks.workunit.client.0.trial112.stderr: { 2026-02-13T02:40:33.648 INFO:tasks.workunit.client.0.trial112.stderr: "pgid": "1.0", 2026-02-13T02:40:33.648 INFO:tasks.workunit.client.0.trial112.stderr: "sched_time": "2026-02-16T02:39:27.514665+0000", 2026-02-13T02:40:33.648 INFO:tasks.workunit.client.0.trial112.stderr: "orig_sched_time": "2026-02-16T02:39:27.514665+0000", 2026-02-13T02:40:33.648 INFO:tasks.workunit.client.0.trial112.stderr: "forced": false, 2026-02-13T02:40:33.649 INFO:tasks.workunit.client.0.trial112.stderr: "level": "shallow", 2026-02-13T02:40:33.649 INFO:tasks.workunit.client.0.trial112.stderr: "urgency": "periodic-regular", 2026-02-13T02:40:33.649 INFO:tasks.workunit.client.0.trial112.stderr: "eligible": false, 2026-02-13T02:40:33.649 INFO:tasks.workunit.client.0.trial112.stderr: "last_issue": "ok" 2026-02-13T02:40:33.649 INFO:tasks.workunit.client.0.trial112.stderr: }, 2026-02-13T02:40:33.649 INFO:tasks.workunit.client.0.trial112.stderr: { 2026-02-13T02:40:33.649 INFO:tasks.workunit.client.0.trial112.stderr: "pgid": "1.0", 2026-02-13T02:40:33.649 INFO:tasks.workunit.client.0.trial112.stderr: "sched_time": "2026-02-18T13:04:07.421830+0000", 2026-02-13T02:40:33.649 INFO:tasks.workunit.client.0.trial112.stderr: "orig_sched_time": "2026-02-18T13:04:07.421830+0000", 2026-02-13T02:40:33.650 INFO:tasks.workunit.client.0.trial112.stderr: "forced": false, 2026-02-13T02:40:33.650 INFO:tasks.workunit.client.0.trial112.stderr: "level": "deep", 2026-02-13T02:40:33.650 INFO:tasks.workunit.client.0.trial112.stderr: "urgency": "periodic-regular", 2026-02-13T02:40:33.650 INFO:tasks.workunit.client.0.trial112.stderr: "eligible": false, 2026-02-13T02:40:33.650 INFO:tasks.workunit.client.0.trial112.stderr: "last_issue": "ok" 2026-02-13T02:40:33.650 INFO:tasks.workunit.client.0.trial112.stderr: } 2026-02-13T02:40:33.650 INFO:tasks.workunit.client.0.trial112.stderr:]' 2026-02-13T02:40:33.651 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2026-02-16T02:39:27.514665+0000",' '"orig_sched_time":' '"2026-02-16T02:39:27.514665+0000",' '"forced":' false, '"level":' '"shallow",' '"urgency":' '"periodic-regular",' '"eligible":' false, '"last_issue":' '"ok"' '},' '{' '"pgid":' '"1.0",' '"sched_time":' '"2026-02-18T13:04:07.421830+0000",' '"orig_sched_time":' '"2026-02-18T13:04:07.421830+0000",' '"forced":' false, '"level":' '"deep",' '"urgency":' '"periodic-regular",' '"eligible":' false, '"last_issue":' '"ok"' '}' ']' 2026-02-13T02:40:33.651 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: jq '.[0].sched_time' 2026-02-13T02:40:33.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: eval 'SCHED_TIME="2026-02-16T02:39:27.514665+0000"' 2026-02-13T02:40:33.659 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: SCHED_TIME=2026-02-16T02:39:27.514665+0000 2026-02-13T02:40:33.659 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: echo 2026-02-16T02:39:27.514665+0000 2026-02-13T02:40:33.659 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2026-02-13T02:40:33.661 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: date +%Y-%m-%d -d 'now + 3 days' 2026-02-13T02:40:33.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: test 2026-02-16 = 2026-02-16 2026-02-13T02:40:33.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:198: TEST_interval_changes: perf_counters td/osd-scrub-test 2 2026-02-13T02:40:33.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2026-02-13T02:40:33.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=2 2026-02-13T02:40:33.664 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 2 - 1 2026-02-13T02:40:33.664 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 1 2026-02-13T02:40:33.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:40:33.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2026-02-13T02:40:33.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-13T02:40:33.736 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:40:33.737 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_ec": [ 2026-02-13T02:40:33.737 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:40:33.737 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:40:33.737 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:40:33.737 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:40:33.737 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:40:33.737 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:40:33.737 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:40:33.737 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:40:33.737 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:40:33.737 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:40:33.738 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:40:33.738 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:40:33.738 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:40:33.738 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:40:33.738 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_repl": [ 2026-02-13T02:40:33.738 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:40:33.738 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:40:33.738 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:40:33.738 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:40:33.738 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:40:33.738 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:40:33.738 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:40:33.739 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:40:33.739 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:40:33.739 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:40:33.739 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:40:33.739 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:40:33.739 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:40:33.739 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:40:33.739 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_ec": [ 2026-02-13T02:40:33.739 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:40:33.739 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:40:33.739 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:40:33.739 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:40:33.739 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:40:33.740 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:40:33.740 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:40:33.740 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:40:33.740 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:40:33.740 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:40:33.740 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:40:33.740 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:40:33.740 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:40:33.740 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:40:33.740 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_repl": [ 2026-02-13T02:40:33.740 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:40:33.740 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:40:33.741 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:40:33.741 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:40:33.741 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:40:33.741 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:40:33.741 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:40:33.741 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:40:33.741 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:40:33.741 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:40:33.741 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:40:33.741 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:40:33.741 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:40:33.741 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:40:33.742 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:40:33.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:40:33.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2026-02-13T02:40:33.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-13T02:40:33.808 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:40:33.808 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_ec": [ 2026-02-13T02:40:33.808 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:40:33.808 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:40:33.808 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:40:33.808 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:40:33.808 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:40:33.809 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:40:33.809 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:40:33.809 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:40:33.809 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:40:33.809 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:40:33.809 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:40:33.809 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:40:33.809 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:40:33.809 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:40:33.809 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_repl": [ 2026-02-13T02:40:33.809 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:40:33.809 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:40:33.810 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:40:33.810 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:40:33.810 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:40:33.810 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:40:33.810 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:40:33.810 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:40:33.810 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:40:33.810 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:40:33.810 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:40:33.810 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:40:33.810 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:40:33.810 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:40:33.810 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_ec": [ 2026-02-13T02:40:33.811 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:40:33.811 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:40:33.811 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:40:33.811 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:40:33.811 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:40:33.811 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:40:33.811 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:40:33.811 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:40:33.811 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:40:33.811 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:40:33.811 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:40:33.811 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:40:33.812 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:40:33.812 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:40:33.812 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_repl": [ 2026-02-13T02:40:33.812 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:40:33.812 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:40:33.812 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:40:33.812 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:40:33.812 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:40:33.812 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:40:33.812 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:40:33.812 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:40:33.812 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:40:33.812 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:40:33.813 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:40:33.813 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:40:33.813 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:40:33.813 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:40:33.813 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:40:33.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_interval_changes ------------------' 2026-02-13T02:40:33.814 INFO:tasks.workunit.client.0.trial112.stdout:-------------- Teardown Test TEST_interval_changes ------------------ 2026-02-13T02:40:33.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2026-02-13T02:40:33.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-13T02:40:33.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:40:33.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-13T02:40:33.815 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:40:33.815 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:40:33.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:40:33.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:40:33.816 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:40:33.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:40:33.918 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:40:33.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:40:33.919 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:40:33.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:40:33.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:40:33.921 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:40:33.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:40:33.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:40:33.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:40:33.922 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:40:33.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:40:33.924 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:40:33.924 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-13T02:40:33.933 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:40:33.933 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:40:33.933 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:40:33.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.508946 2026-02-13T02:40:33.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:40:33.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:40:33.935 INFO:tasks.workunit.client.0.trial112.stdout:-------------- Complete Test TEST_interval_changes ------------------ 2026-02-13T02:40:33.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_interval_changes ------------------' 2026-02-13T02:40:33.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2026-02-13T02:40:33.936 INFO:tasks.workunit.client.0.trial112.stdout:-------------- Prepare Test TEST_just_deep_scrubs ------------------- 2026-02-13T02:40:33.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_just_deep_scrubs -------------------' 2026-02-13T02:40:33.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2026-02-13T02:40:33.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2026-02-13T02:40:33.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2026-02-13T02:40:33.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-13T02:40:33.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:40:33.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-13T02:40:33.938 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:40:33.938 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:40:33.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:40:33.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:40:33.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:40:33.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:40:33.939 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:40:33.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:40:33.940 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:40:33.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:40:33.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:40:33.942 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:40:33.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:40:33.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:40:33.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:40:33.944 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:40:33.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:40:33.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:40:33.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-13T02:40:33.947 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:40:33.947 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:40:33.947 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:40:33.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.508946 2026-02-13T02:40:33.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:40:33.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:40:33.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2026-02-13T02:40:33.950 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T02:40:33.950 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:40:33.950 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:40:33.950 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.508946 2026-02-13T02:40:33.951 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T02:40:33.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-13T02:40:33.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T02:40:33.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2026-02-13T02:40:33.953 INFO:tasks.workunit.client.0.trial112.stdout:-------------- Run Test TEST_just_deep_scrubs ----------------------- 2026-02-13T02:40:33.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_just_deep_scrubs -----------------------' 2026-02-13T02:40:33.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_just_deep_scrubs td/osd-scrub-test 2026-02-13T02:40:33.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:462: TEST_just_deep_scrubs: local dir=td/osd-scrub-test 2026-02-13T02:40:33.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:463: TEST_just_deep_scrubs: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='4' ['pool_name']='test') 2026-02-13T02:40:33.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:463: TEST_just_deep_scrubs: local -A cluster_conf 2026-02-13T02:40:33.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:469: TEST_just_deep_scrubs: standard_scrub_cluster td/osd-scrub-test cluster_conf 2026-02-13T02:40:33.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local dir=td/osd-scrub-test 2026-02-13T02:40:33.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:230: standard_scrub_cluster: local -n args=cluster_conf 2026-02-13T02:40:33.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local OSDS=3 2026-02-13T02:40:33.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: local pg_num=4 2026-02-13T02:40:33.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:235: standard_scrub_cluster: local poolname=test 2026-02-13T02:40:33.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:236: standard_scrub_cluster: args['pool_name']=test 2026-02-13T02:40:33.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:238: standard_scrub_cluster: local pool_default_size=3 2026-02-13T02:40:33.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:239: standard_scrub_cluster: args['pool_default_size']=3 2026-02-13T02:40:33.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:241: standard_scrub_cluster: local extra_pars= 2026-02-13T02:40:33.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:242: standard_scrub_cluster: local debug_msg=dbg 2026-02-13T02:40:33.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:245: standard_scrub_cluster: local saved_echo_flag=x 2026-02-13T02:40:33.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:246: standard_scrub_cluster: set +x 2026-02-13T02:40:34.232 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 15e4f1f3-9dec-4786-8f9d-df042c5c49a4 2026-02-13T02:40:34.335 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:40:34.362 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:40:34.354+0000 7f9184500a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:40:34.364 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:40:34.358+0000 7f9184500a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:40:34.365 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:40:34.358+0000 7f9184500a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:40:34.365 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:40:34.358+0000 7f9184500a40 -1 bdev(0x5652c339cc00 td/osd-scrub-test/0/block) open stat got: (1) Operation not permitted 2026-02-13T02:40:34.366 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:40:34.358+0000 7f9184500a40 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2026-02-13T02:40:35.064 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T02:40:35.167 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T02:40:35.185 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:40:35.178+0000 7fe27d43aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:40:35.187 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:40:35.182+0000 7fe27d43aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:40:35.188 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:40:35.182+0000 7fe27d43aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:40:35.258 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:40:35.681 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:40:35.674+0000 7fe27d43aa40 -1 Falling back to public interface 2026-02-13T02:40:36.348 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:40:36.660 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:40:36.654+0000 7fe27d43aa40 -1 osd.0 0 log_to_monitors true 2026-02-13T02:40:37.487 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:40:37.648 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3831124147,v1:127.0.0.1:6801/3831124147] [v2:127.0.0.1:6802/3831124147,v1:127.0.0.1:6803/3831124147] exists,up 15e4f1f3-9dec-4786-8f9d-df042c5c49a4 2026-02-13T02:40:37.653 INFO:tasks.workunit.client.0.trial112.stdout:add osd1 c2aaa41f-b29d-443e-af01-05f0701af3cc 2026-02-13T02:40:37.834 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:40:37.863 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:40:37.858+0000 7f9dc7b1ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:40:37.865 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:40:37.858+0000 7f9dc7b1ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:40:37.866 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:40:37.858+0000 7f9dc7b1ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:40:37.866 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:40:37.858+0000 7f9dc7b1ea40 -1 bdev(0x555d5d542c00 td/osd-scrub-test/1/block) open stat got: (1) Operation not permitted 2026-02-13T02:40:37.866 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:40:37.858+0000 7f9dc7b1ea40 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2026-02-13T02:40:38.511 INFO:tasks.workunit.client.0.trial112.stdout:adding osd1 key to auth repository 2026-02-13T02:40:38.721 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T02:40:38.751 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:40:38.746+0000 7fbad2185a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:40:38.753 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:40:38.746+0000 7fbad2185a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:40:38.754 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:40:38.746+0000 7fbad2185a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:40:38.875 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:40:39.128 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:40:39.122+0000 7fbad2185a40 -1 Falling back to public interface 2026-02-13T02:40:39.710 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:40:39.702+0000 7fbad2185a40 -1 osd.1 0 log_to_monitors true 2026-02-13T02:40:40.039 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:40:41.198 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:40:41.359 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3078014031,v1:127.0.0.1:6811/3078014031] [v2:127.0.0.1:6812/3078014031,v1:127.0.0.1:6813/3078014031] exists,up c2aaa41f-b29d-443e-af01-05f0701af3cc 2026-02-13T02:40:41.364 INFO:tasks.workunit.client.0.trial112.stdout:add osd2 4f22c3c6-c5e5-4847-b76f-7f680db15fcf 2026-02-13T02:40:41.535 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:40:41.563 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:40:41.558+0000 7f34b86c9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:40:41.564 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:40:41.558+0000 7f34b86c9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:40:41.565 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:40:41.558+0000 7f34b86c9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:40:41.566 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:40:41.558+0000 7f34b86c9a40 -1 bdev(0x558026e68c00 td/osd-scrub-test/2/block) open stat got: (1) Operation not permitted 2026-02-13T02:40:41.566 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:40:41.558+0000 7f34b86c9a40 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2026-02-13T02:40:42.214 INFO:tasks.workunit.client.0.trial112.stdout:adding osd2 key to auth repository 2026-02-13T02:40:42.416 INFO:tasks.workunit.client.0.trial112.stdout:start osd.2 2026-02-13T02:40:42.444 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:40:42.438+0000 7efd4a688a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:40:42.446 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:40:42.438+0000 7efd4a688a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:40:42.447 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:40:42.442+0000 7efd4a688a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:40:42.577 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:40:42.868 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:40:42.862+0000 7efd4a688a40 -1 Falling back to public interface 2026-02-13T02:40:43.513 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:40:43.506+0000 7efd4a688a40 -1 osd.2 0 log_to_monitors true 2026-02-13T02:40:43.738 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:40:44.904 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:40:45.059 INFO:tasks.workunit.client.0.trial112.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1205169225,v1:127.0.0.1:6819/1205169225] [v2:127.0.0.1:6820/1205169225,v1:127.0.0.1:6821/1205169225] exists,up 4f22c3c6-c5e5-4847-b76f-7f680db15fcf 2026-02-13T02:40:45.265 INFO:tasks.workunit.client.0.trial112.stderr:pool 'test' created 2026-02-13T02:40:46.770 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836490 2026-02-13T02:40:49.268 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 42949672967 2026-02-13T02:40:49.431 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 64424509444 2026-02-13T02:40:51.288 INFO:tasks.workunit.client.0.trial112.stdout:standard_scrub_cluster: dbg: test pool is test 1 2026-02-13T02:40:51.289 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:470: TEST_just_deep_scrubs: local poolid=1 2026-02-13T02:40:51.289 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:471: TEST_just_deep_scrubs: local poolname=test 2026-02-13T02:40:51.289 INFO:tasks.workunit.client.0.trial112.stdout:Pool: test : 1 2026-02-13T02:40:51.290 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:472: TEST_just_deep_scrubs: echo 'Pool: test : 1' 2026-02-13T02:40:51.290 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:474: TEST_just_deep_scrubs: TESTDATA=testdata.508946 2026-02-13T02:40:51.290 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:475: TEST_just_deep_scrubs: local objects=90 2026-02-13T02:40:51.291 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:476: TEST_just_deep_scrubs: dd if=/dev/urandom of=testdata.508946 bs=1032 count=1 2026-02-13T02:40:51.291 INFO:tasks.workunit.client.0.trial112.stderr:1+0 records in 2026-02-13T02:40:51.291 INFO:tasks.workunit.client.0.trial112.stderr:1+0 records out 2026-02-13T02:40:51.291 INFO:tasks.workunit.client.0.trial112.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 6.1084e-05 s, 16.9 MB/s 2026-02-13T02:40:51.291 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: seq 1 90 2026-02-13T02:40:51.291 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:51.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj1 testdata.508946 2026-02-13T02:40:51.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:51.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj2 testdata.508946 2026-02-13T02:40:51.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:51.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj3 testdata.508946 2026-02-13T02:40:51.358 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:51.359 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj4 testdata.508946 2026-02-13T02:40:51.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:51.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj5 testdata.508946 2026-02-13T02:40:51.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:51.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj6 testdata.508946 2026-02-13T02:40:51.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:51.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj7 testdata.508946 2026-02-13T02:40:51.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:51.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj8 testdata.508946 2026-02-13T02:40:51.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:51.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj9 testdata.508946 2026-02-13T02:40:51.497 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:51.497 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj10 testdata.508946 2026-02-13T02:40:51.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:51.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj11 testdata.508946 2026-02-13T02:40:51.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:51.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj12 testdata.508946 2026-02-13T02:40:51.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:51.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj13 testdata.508946 2026-02-13T02:40:51.594 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:51.594 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj14 testdata.508946 2026-02-13T02:40:51.618 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:51.618 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj15 testdata.508946 2026-02-13T02:40:51.642 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:51.642 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj16 testdata.508946 2026-02-13T02:40:51.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:51.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj17 testdata.508946 2026-02-13T02:40:51.692 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:51.692 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj18 testdata.508946 2026-02-13T02:40:51.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:51.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj19 testdata.508946 2026-02-13T02:40:51.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:51.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj20 testdata.508946 2026-02-13T02:40:51.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:51.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj21 testdata.508946 2026-02-13T02:40:51.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:51.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj22 testdata.508946 2026-02-13T02:40:51.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:51.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj23 testdata.508946 2026-02-13T02:40:51.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:51.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj24 testdata.508946 2026-02-13T02:40:51.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:51.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj25 testdata.508946 2026-02-13T02:40:51.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:51.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj26 testdata.508946 2026-02-13T02:40:51.904 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:51.904 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj27 testdata.508946 2026-02-13T02:40:51.928 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:51.928 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj28 testdata.508946 2026-02-13T02:40:51.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:51.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj29 testdata.508946 2026-02-13T02:40:51.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:51.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj30 testdata.508946 2026-02-13T02:40:52.001 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj31 testdata.508946 2026-02-13T02:40:52.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj32 testdata.508946 2026-02-13T02:40:52.050 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.050 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj33 testdata.508946 2026-02-13T02:40:52.074 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.074 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj34 testdata.508946 2026-02-13T02:40:52.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj35 testdata.508946 2026-02-13T02:40:52.122 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.122 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj36 testdata.508946 2026-02-13T02:40:52.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj37 testdata.508946 2026-02-13T02:40:52.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.171 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj38 testdata.508946 2026-02-13T02:40:52.196 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.196 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj39 testdata.508946 2026-02-13T02:40:52.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj40 testdata.508946 2026-02-13T02:40:52.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj41 testdata.508946 2026-02-13T02:40:52.270 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.271 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj42 testdata.508946 2026-02-13T02:40:52.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.289 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj43 testdata.508946 2026-02-13T02:40:52.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj44 testdata.508946 2026-02-13T02:40:52.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj45 testdata.508946 2026-02-13T02:40:52.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj46 testdata.508946 2026-02-13T02:40:52.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj47 testdata.508946 2026-02-13T02:40:52.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj48 testdata.508946 2026-02-13T02:40:52.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj49 testdata.508946 2026-02-13T02:40:52.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj50 testdata.508946 2026-02-13T02:40:52.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj51 testdata.508946 2026-02-13T02:40:52.494 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.494 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj52 testdata.508946 2026-02-13T02:40:52.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj53 testdata.508946 2026-02-13T02:40:52.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj54 testdata.508946 2026-02-13T02:40:52.568 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.568 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj55 testdata.508946 2026-02-13T02:40:52.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj56 testdata.508946 2026-02-13T02:40:52.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.618 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj57 testdata.508946 2026-02-13T02:40:52.643 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.643 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj58 testdata.508946 2026-02-13T02:40:52.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj59 testdata.508946 2026-02-13T02:40:52.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj60 testdata.508946 2026-02-13T02:40:52.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj61 testdata.508946 2026-02-13T02:40:52.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj62 testdata.508946 2026-02-13T02:40:52.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj63 testdata.508946 2026-02-13T02:40:52.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj64 testdata.508946 2026-02-13T02:40:52.806 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj65 testdata.508946 2026-02-13T02:40:52.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj66 testdata.508946 2026-02-13T02:40:52.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj67 testdata.508946 2026-02-13T02:40:52.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj68 testdata.508946 2026-02-13T02:40:52.905 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.905 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj69 testdata.508946 2026-02-13T02:40:52.929 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj70 testdata.508946 2026-02-13T02:40:52.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj71 testdata.508946 2026-02-13T02:40:52.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj72 testdata.508946 2026-02-13T02:40:52.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:52.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj73 testdata.508946 2026-02-13T02:40:53.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:53.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj74 testdata.508946 2026-02-13T02:40:53.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:53.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj75 testdata.508946 2026-02-13T02:40:53.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:53.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj76 testdata.508946 2026-02-13T02:40:53.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:53.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj77 testdata.508946 2026-02-13T02:40:53.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:53.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj78 testdata.508946 2026-02-13T02:40:53.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:53.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj79 testdata.508946 2026-02-13T02:40:53.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:53.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj80 testdata.508946 2026-02-13T02:40:53.190 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:53.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj81 testdata.508946 2026-02-13T02:40:53.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:53.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj82 testdata.508946 2026-02-13T02:40:53.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:53.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj83 testdata.508946 2026-02-13T02:40:53.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:53.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj84 testdata.508946 2026-02-13T02:40:53.277 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:53.277 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj85 testdata.508946 2026-02-13T02:40:53.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:53.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj86 testdata.508946 2026-02-13T02:40:53.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:53.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj87 testdata.508946 2026-02-13T02:40:53.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:53.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj88 testdata.508946 2026-02-13T02:40:53.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:53.374 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj89 testdata.508946 2026-02-13T02:40:53.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-13T02:40:53.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj90 testdata.508946 2026-02-13T02:40:53.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:481: TEST_just_deep_scrubs: rm -f testdata.508946 2026-02-13T02:40:53.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:486: TEST_just_deep_scrubs: ceph tell 'osd.*' config set osd_scrub_retry_after_noscrub 2 2026-02-13T02:40:53.488 INFO:tasks.workunit.client.0.trial112.stdout:osd.0: { 2026-02-13T02:40:53.489 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_retry_after_noscrub = '' (not observed, change may require restart) " 2026-02-13T02:40:53.489 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:40:53.496 INFO:tasks.workunit.client.0.trial112.stdout:osd.1: { 2026-02-13T02:40:53.496 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_retry_after_noscrub = '' (not observed, change may require restart) " 2026-02-13T02:40:53.496 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:40:53.503 INFO:tasks.workunit.client.0.trial112.stdout:osd.2: { 2026-02-13T02:40:53.503 INFO:tasks.workunit.client.0.trial112.stdout: "success": "osd_scrub_retry_after_noscrub = '' (not observed, change may require restart) " 2026-02-13T02:40:53.503 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:40:53.511 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:487: TEST_just_deep_scrubs: ceph osd set noscrub 2026-02-13T02:40:53.726 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:40:53.736 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:488: TEST_just_deep_scrubs: ceph osd set nodeep-scrub 2026-02-13T02:40:53.931 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is set 2026-02-13T02:40:53.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:489: TEST_just_deep_scrubs: sleep 6 2026-02-13T02:40:59.945 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:490: TEST_just_deep_scrubs: date -Ins 2026-02-13T02:40:59.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:490: TEST_just_deep_scrubs: local now_is=2026-02-13T02:40:59,942099388+00:00 2026-02-13T02:40:59.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:491: TEST_just_deep_scrubs: declare -A sched_data 2026-02-13T02:40:59.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:492: TEST_just_deep_scrubs: local pgid=1.2 2026-02-13T02:40:59.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:495: TEST_just_deep_scrubs: set_query_debug 1.2 2026-02-13T02:40:59.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:417: set_query_debug: local pgid=1.2 2026-02-13T02:40:59.947 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:419: set_query_debug: ceph pg dump pgs_brief 2026-02-13T02:40:59.948 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:419: set_query_debug: awk -v 'pg=^1.2' -n -e '$0 ~ pg { print(gensub(/[^0-9]*([0-9]+).*/,"\\1","g",$5)); }' 2026-02-13T02:41:00.096 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs_brief 2026-02-13T02:41:00.107 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:419: set_query_debug: local prim_osd=0 2026-02-13T02:41:00.107 INFO:tasks.workunit.client.0.trial112.stdout:Setting scrub debug data. Primary for 1.2 is 0 2026-02-13T02:41:00.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:421: set_query_debug: echo 'Setting scrub debug data. Primary for 1.2 is 0' 2026-02-13T02:41:00.109 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:422: set_query_debug: get_asok_path osd.0 2026-02-13T02:41:00.109 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-13T02:41:00.109 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-13T02:41:00.109 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:41:00.109 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:41:00.110 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:41:00.110 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.508946/ceph-osd.0.asok 2026-02-13T02:41:00.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:422: set_query_debug: CEPH_ARGS= 2026-02-13T02:41:00.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:422: set_query_debug: ceph --format=json daemon /tmp/ceph-asok.508946/ceph-osd.0.asok scrubdebug 1.2 set sessions 2026-02-13T02:41:00.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:497: TEST_just_deep_scrubs: extract_published_sch 1.2 2026-02-13T02:40:59,942099388+00:00 2026-02-13T02:40:59,942099388+00:00 sched_data 2026-02-13T02:41:00.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:17: extract_published_sch: local pgn=1.2 2026-02-13T02:41:00.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:18: extract_published_sch: local -n dict=sched_data 2026-02-13T02:41:00.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:19: extract_published_sch: local current_time=2026-02-13T02:40:59,942099388+00:00 2026-02-13T02:41:00.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:20: extract_published_sch: local extra_time=2026-02-13T02:40:59,942099388+00:00 2026-02-13T02:41:00.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:21: extract_published_sch: local extr_dbg=2 2026-02-13T02:41:00.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:24: extract_published_sch: local saved_echo_flag=x 2026-02-13T02:41:00.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:25: extract_published_sch: set +x 2026-02-13T02:41:00.320 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:41:00.331 INFO:tasks.workunit.client.0.trial112.stdout:{"success":true}from pg dump pg: { 2026-02-13T02:41:00.331 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_pg_state": "active+clean", 2026-02-13T02:41:00.331 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_state_has_scrubbing": false, 2026-02-13T02:41:00.331 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_last_duration": 0, 2026-02-13T02:41:00.332 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_schedule": "periodic scrub scheduled", 2026-02-13T02:41:00.332 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_schedule_at": "2026-02-14T02:40:45.260162+0000", 2026-02-13T02:41:00.332 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_is_future": true, 2026-02-13T02:41:00.332 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_vs_date": true, 2026-02-13T02:41:00.332 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_reported_epoch": 21, 2026-02-13T02:41:00.332 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_seq": 42 2026-02-13T02:41:00.332 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:41:00.332 INFO:tasks.workunit.client.0.trial112.stdout:query output: 2026-02-13T02:41:00.405 INFO:tasks.workunit.client.0.trial112.stdout: "scrubber": { 2026-02-13T02:41:00.405 INFO:tasks.workunit.client.0.trial112.stdout: "active": false, 2026-02-13T02:41:00.405 INFO:tasks.workunit.client.0.trial112.stdout: "must_scrub": false, 2026-02-13T02:41:00.405 INFO:tasks.workunit.client.0.trial112.stdout: "must_deep_scrub": false, 2026-02-13T02:41:00.405 INFO:tasks.workunit.client.0.trial112.stdout: "must_repair": false, 2026-02-13T02:41:00.405 INFO:tasks.workunit.client.0.trial112.stdout: "scrub_reg_stamp": "2026-02-14T02:40:45.260162+0000", 2026-02-13T02:41:00.406 INFO:tasks.workunit.client.0.trial112.stdout: "schedule": "scrub scheduled @ 2026-02-14T02:40:45.260 (2026-02-14T02:40:45.260)", 2026-02-13T02:41:00.406 INFO:tasks.workunit.client.0.trial112.stdout: "test_sequence": 2826240 2026-02-13T02:41:00.406 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:41:00.406 INFO:tasks.workunit.client.0.trial112.stdout: "agent_state": {} 2026-02-13T02:41:00.489 INFO:tasks.workunit.client.0.trial112.stdout:( 2026-02-13T02:41:00.489 INFO:tasks.workunit.client.0.trial112.stdout:[query_epoch]=21 2026-02-13T02:41:00.490 INFO:tasks.workunit.client.0.trial112.stdout:[query_seq]=43 2026-02-13T02:41:00.490 INFO:tasks.workunit.client.0.trial112.stdout:[query_active]=false 2026-02-13T02:41:00.490 INFO:tasks.workunit.client.0.trial112.stdout:[query_schedule]='scrub scheduled' 2026-02-13T02:41:00.490 INFO:tasks.workunit.client.0.trial112.stdout:[query_schedule_at]='2026-02-14T02:40:45.260' 2026-02-13T02:41:00.490 INFO:tasks.workunit.client.0.trial112.stdout:[query_target_at]='2026-02-14T02:40:45.260' 2026-02-13T02:41:00.490 INFO:tasks.workunit.client.0.trial112.stdout:[query_last_duration]=0 2026-02-13T02:41:00.490 INFO:tasks.workunit.client.0.trial112.stdout:[query_last_stamp]='2026-02-13T02:40:45.260162+0000' 2026-02-13T02:41:00.490 INFO:tasks.workunit.client.0.trial112.stdout:[query_last_scrub]='0x0' 2026-02-13T02:41:00.490 INFO:tasks.workunit.client.0.trial112.stdout:[query_is_future]=true 2026-02-13T02:41:00.490 INFO:tasks.workunit.client.0.trial112.stdout:[query_vs_date]=true 2026-02-13T02:41:00.490 INFO:tasks.workunit.client.0.trial112.stdout:[query_scrub_seq]=2826240 2026-02-13T02:41:00.490 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_pg_state]='active+clean' 2026-02-13T02:41:00.491 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_state_has_scrubbing]=false 2026-02-13T02:41:00.491 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_last_duration]=0 2026-02-13T02:41:00.491 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_schedule]='periodic scrub scheduled' 2026-02-13T02:41:00.491 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_schedule_at]='2026-02-14T02:40:45.260162+0000' 2026-02-13T02:41:00.491 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_is_future]=true 2026-02-13T02:41:00.491 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_vs_date]=true 2026-02-13T02:41:00.491 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_reported_epoch]=21 2026-02-13T02:41:00.491 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_seq]=42 2026-02-13T02:41:00.491 INFO:tasks.workunit.client.0.trial112.stdout:) 2026-02-13T02:41:00.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:498: TEST_just_deep_scrubs: local saved_last_stamp=2026-02-13T02:40:45.260162+0000 2026-02-13T02:41:00.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:499: TEST_just_deep_scrubs: local dbg_counter_at_start=2826240 2026-02-13T02:41:00.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:500: TEST_just_deep_scrubs: echo 'test counter @ start: 2826240' 2026-02-13T02:41:00.501 INFO:tasks.workunit.client.0.trial112.stdout:test counter @ start: 2826240 2026-02-13T02:41:00.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:502: TEST_just_deep_scrubs: ceph tell 1.2 schedule-deep-scrub 2026-02-13T02:41:00.566 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:41:00.566 INFO:tasks.workunit.client.0.trial112.stdout: "deep": true, 2026-02-13T02:41:00.566 INFO:tasks.workunit.client.0.trial112.stdout: "must": false, 2026-02-13T02:41:00.566 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "2026-02-01T21:52:50.561959+0000" 2026-02-13T02:41:00.566 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:41:00.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:504: TEST_just_deep_scrubs: sleep 5 2026-02-13T02:41:05.575 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:506: TEST_just_deep_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-13T02:41:05.576 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:506: TEST_just_deep_scrubs: jq -r '.pg_stats[] | "\(.pgid) \(.stat_sum.num_objects)"' 2026-02-13T02:41:05.730 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:41:05.739 INFO:tasks.workunit.client.0.trial112.stdout:1.3 20 2026-02-13T02:41:05.739 INFO:tasks.workunit.client.0.trial112.stdout:1.2 20 2026-02-13T02:41:05.739 INFO:tasks.workunit.client.0.trial112.stdout:1.1 25 2026-02-13T02:41:05.739 INFO:tasks.workunit.client.0.trial112.stdout:1.0 25 2026-02-13T02:41:05.740 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:507: TEST_just_deep_scrubs: ceph pg 1.2 query --format=json-pretty 2026-02-13T02:41:05.740 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:507: TEST_just_deep_scrubs: jq -r .info.stats.stat_sum.num_objects 2026-02-13T02:41:05.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:507: TEST_just_deep_scrubs: echo 'Objects # in pg 1.2: ' 20 2026-02-13T02:41:05.813 INFO:tasks.workunit.client.0.trial112.stdout:Objects # in pg 1.2: 20 2026-02-13T02:41:05.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:509: TEST_just_deep_scrubs: declare -A sc_data_2 2026-02-13T02:41:05.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:510: TEST_just_deep_scrubs: extract_published_sch 1.2 2026-02-13T02:40:59,942099388+00:00 2026-02-13T02:40:59,942099388+00:00 sc_data_2 2026-02-13T02:41:05.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:17: extract_published_sch: local pgn=1.2 2026-02-13T02:41:05.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:18: extract_published_sch: local -n dict=sc_data_2 2026-02-13T02:41:05.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:19: extract_published_sch: local current_time=2026-02-13T02:40:59,942099388+00:00 2026-02-13T02:41:05.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:20: extract_published_sch: local extra_time=2026-02-13T02:40:59,942099388+00:00 2026-02-13T02:41:05.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:21: extract_published_sch: local extr_dbg=2 2026-02-13T02:41:05.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:24: extract_published_sch: local saved_echo_flag=x 2026-02-13T02:41:05.815 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:25: extract_published_sch: set +x 2026-02-13T02:41:05.962 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:41:05.972 INFO:tasks.workunit.client.0.trial112.stdout:from pg dump pg: { 2026-02-13T02:41:05.972 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_pg_state": "active+clean", 2026-02-13T02:41:05.972 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_state_has_scrubbing": false, 2026-02-13T02:41:05.972 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_last_duration": 0, 2026-02-13T02:41:05.972 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_schedule": "periodic deep scrub scheduled", 2026-02-13T02:41:05.972 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_schedule_at": "2026-02-13T02:41:02.722808+0000", 2026-02-13T02:41:05.972 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_is_future": true, 2026-02-13T02:41:05.972 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_vs_date": true, 2026-02-13T02:41:05.972 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_reported_epoch": 21, 2026-02-13T02:41:05.973 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_seq": 47 2026-02-13T02:41:05.973 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:41:05.973 INFO:tasks.workunit.client.0.trial112.stdout:query output: 2026-02-13T02:41:06.046 INFO:tasks.workunit.client.0.trial112.stdout: "scrubber": { 2026-02-13T02:41:06.046 INFO:tasks.workunit.client.0.trial112.stdout: "active": false, 2026-02-13T02:41:06.046 INFO:tasks.workunit.client.0.trial112.stdout: "must_scrub": false, 2026-02-13T02:41:06.047 INFO:tasks.workunit.client.0.trial112.stdout: "must_deep_scrub": false, 2026-02-13T02:41:06.047 INFO:tasks.workunit.client.0.trial112.stdout: "must_repair": false, 2026-02-13T02:41:06.047 INFO:tasks.workunit.client.0.trial112.stdout: "scrub_reg_stamp": "2026-02-13T02:41:05.696803+0000", 2026-02-13T02:41:06.047 INFO:tasks.workunit.client.0.trial112.stdout: "schedule": "queued for deep scrub at 2026-02-06T06:16:25.774", 2026-02-13T02:41:06.047 INFO:tasks.workunit.client.0.trial112.stdout: "test_sequence": 2826240 2026-02-13T02:41:06.047 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:41:06.047 INFO:tasks.workunit.client.0.trial112.stdout: "agent_state": {} 2026-02-13T02:41:06.133 INFO:tasks.workunit.client.0.trial112.stdout:( 2026-02-13T02:41:06.134 INFO:tasks.workunit.client.0.trial112.stdout:[query_epoch]=21 2026-02-13T02:41:06.134 INFO:tasks.workunit.client.0.trial112.stdout:[query_seq]=48 2026-02-13T02:41:06.134 INFO:tasks.workunit.client.0.trial112.stdout:[query_active]=false 2026-02-13T02:41:06.134 INFO:tasks.workunit.client.0.trial112.stdout:[query_schedule]='queued for deep scrub at 2026-02-06T06:16:25.774' 2026-02-13T02:41:06.134 INFO:tasks.workunit.client.0.trial112.stdout:[query_schedule_at]='0' 2026-02-13T02:41:06.134 INFO:tasks.workunit.client.0.trial112.stdout:[query_target_at]='0' 2026-02-13T02:41:06.134 INFO:tasks.workunit.client.0.trial112.stdout:[query_last_duration]=0 2026-02-13T02:41:06.134 INFO:tasks.workunit.client.0.trial112.stdout:[query_last_stamp]='2026-02-13T02:40:45.260162+0000' 2026-02-13T02:41:06.134 INFO:tasks.workunit.client.0.trial112.stdout:[query_last_scrub]='0x0' 2026-02-13T02:41:06.135 INFO:tasks.workunit.client.0.trial112.stdout:[query_is_future]=false 2026-02-13T02:41:06.135 INFO:tasks.workunit.client.0.trial112.stdout:[query_vs_date]=false 2026-02-13T02:41:06.135 INFO:tasks.workunit.client.0.trial112.stdout:[query_scrub_seq]=2826240 2026-02-13T02:41:06.135 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_pg_state]='active+clean' 2026-02-13T02:41:06.135 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_state_has_scrubbing]=false 2026-02-13T02:41:06.135 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_last_duration]=0 2026-02-13T02:41:06.135 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_schedule]='periodic deep scrub scheduled' 2026-02-13T02:41:06.135 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_schedule_at]='2026-02-13T02:41:02.722808+0000' 2026-02-13T02:41:06.135 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_is_future]=true 2026-02-13T02:41:06.135 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_vs_date]=true 2026-02-13T02:41:06.135 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_reported_epoch]=21 2026-02-13T02:41:06.135 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_seq]=47 2026-02-13T02:41:06.135 INFO:tasks.workunit.client.0.trial112.stdout:) 2026-02-13T02:41:06.144 INFO:tasks.workunit.client.0.trial112.stdout:test counter @ should show no change: 2826240 2026-02-13T02:41:06.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:511: TEST_just_deep_scrubs: echo 'test counter @ should show no change: ' 2826240 2026-02-13T02:41:06.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:512: TEST_just_deep_scrubs: (( 0 == 0 )) 2026-02-13T02:41:06.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:513: TEST_just_deep_scrubs: (( 2826240 == 2826240 )) 2026-02-13T02:41:06.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:516: TEST_just_deep_scrubs: ceph osd unset nodeep-scrub 2026-02-13T02:41:06.382 INFO:tasks.workunit.client.0.trial112.stderr:nodeep-scrub is unset 2026-02-13T02:41:06.395 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:517: TEST_just_deep_scrubs: sleep 8 2026-02-13T02:41:14.397 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:518: TEST_just_deep_scrubs: expct_qry_duration=(['query_last_duration']='0' ['query_last_duration_neg']='not0') 2026-02-13T02:41:14.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:518: TEST_just_deep_scrubs: declare -A expct_qry_duration 2026-02-13T02:41:14.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:519: TEST_just_deep_scrubs: sc_data_2=() 2026-02-13T02:41:14.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:520: TEST_just_deep_scrubs: extract_published_sch 1.2 2026-02-13T02:40:59,942099388+00:00 2026-02-13T02:40:59,942099388+00:00 sc_data_2 2026-02-13T02:41:14.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:17: extract_published_sch: local pgn=1.2 2026-02-13T02:41:14.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:18: extract_published_sch: local -n dict=sc_data_2 2026-02-13T02:41:14.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:19: extract_published_sch: local current_time=2026-02-13T02:40:59,942099388+00:00 2026-02-13T02:41:14.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:20: extract_published_sch: local extra_time=2026-02-13T02:40:59,942099388+00:00 2026-02-13T02:41:14.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:21: extract_published_sch: local extr_dbg=2 2026-02-13T02:41:14.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:24: extract_published_sch: local saved_echo_flag=x 2026-02-13T02:41:14.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:25: extract_published_sch: set +x 2026-02-13T02:41:14.543 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:41:14.553 INFO:tasks.workunit.client.0.trial112.stdout:from pg dump pg: { 2026-02-13T02:41:14.553 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_pg_state": "active+clean", 2026-02-13T02:41:14.553 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_state_has_scrubbing": false, 2026-02-13T02:41:14.554 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_last_duration": 1, 2026-02-13T02:41:14.554 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_schedule": "periodic scrub scheduled", 2026-02-13T02:41:14.554 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_schedule_at": "2026-02-14T02:41:06.676139+0000", 2026-02-13T02:41:14.554 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_is_future": true, 2026-02-13T02:41:14.554 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_vs_date": true, 2026-02-13T02:41:14.554 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_reported_epoch": 22, 2026-02-13T02:41:14.554 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_seq": 60 2026-02-13T02:41:14.554 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:41:14.554 INFO:tasks.workunit.client.0.trial112.stdout:query output: 2026-02-13T02:41:14.626 INFO:tasks.workunit.client.0.trial112.stdout: "scrubber": { 2026-02-13T02:41:14.626 INFO:tasks.workunit.client.0.trial112.stdout: "active": false, 2026-02-13T02:41:14.626 INFO:tasks.workunit.client.0.trial112.stdout: "must_scrub": false, 2026-02-13T02:41:14.626 INFO:tasks.workunit.client.0.trial112.stdout: "must_deep_scrub": false, 2026-02-13T02:41:14.626 INFO:tasks.workunit.client.0.trial112.stdout: "must_repair": false, 2026-02-13T02:41:14.626 INFO:tasks.workunit.client.0.trial112.stdout: "scrub_reg_stamp": "2026-02-14T02:41:06.676139+0000", 2026-02-13T02:41:14.626 INFO:tasks.workunit.client.0.trial112.stdout: "schedule": "scrub scheduled @ 2026-02-14T02:41:06.676 (2026-02-14T02:41:06.676)", 2026-02-13T02:41:14.626 INFO:tasks.workunit.client.0.trial112.stdout: "test_sequence": 2826243 2026-02-13T02:41:14.626 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:41:14.626 INFO:tasks.workunit.client.0.trial112.stdout: "agent_state": {} 2026-02-13T02:41:14.711 INFO:tasks.workunit.client.0.trial112.stdout:( 2026-02-13T02:41:14.711 INFO:tasks.workunit.client.0.trial112.stdout:[query_epoch]=22 2026-02-13T02:41:14.711 INFO:tasks.workunit.client.0.trial112.stdout:[query_seq]=61 2026-02-13T02:41:14.711 INFO:tasks.workunit.client.0.trial112.stdout:[query_active]=false 2026-02-13T02:41:14.711 INFO:tasks.workunit.client.0.trial112.stdout:[query_schedule]='scrub scheduled' 2026-02-13T02:41:14.711 INFO:tasks.workunit.client.0.trial112.stdout:[query_schedule_at]='2026-02-14T02:41:06.676' 2026-02-13T02:41:14.711 INFO:tasks.workunit.client.0.trial112.stdout:[query_target_at]='2026-02-14T02:41:06.676' 2026-02-13T02:41:14.711 INFO:tasks.workunit.client.0.trial112.stdout:[query_last_duration]=1 2026-02-13T02:41:14.711 INFO:tasks.workunit.client.0.trial112.stdout:[query_last_stamp]='2026-02-13T02:41:06.676139+0000' 2026-02-13T02:41:14.711 INFO:tasks.workunit.client.0.trial112.stdout:[query_last_scrub]='19x20' 2026-02-13T02:41:14.712 INFO:tasks.workunit.client.0.trial112.stdout:[query_is_future]=true 2026-02-13T02:41:14.712 INFO:tasks.workunit.client.0.trial112.stdout:[query_vs_date]=true 2026-02-13T02:41:14.712 INFO:tasks.workunit.client.0.trial112.stdout:[query_scrub_seq]=2826243 2026-02-13T02:41:14.712 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_pg_state]='active+clean' 2026-02-13T02:41:14.712 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_state_has_scrubbing]=false 2026-02-13T02:41:14.712 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_last_duration]=1 2026-02-13T02:41:14.712 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_schedule]='periodic scrub scheduled' 2026-02-13T02:41:14.712 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_schedule_at]='2026-02-14T02:41:06.676139+0000' 2026-02-13T02:41:14.712 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_is_future]=true 2026-02-13T02:41:14.712 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_vs_date]=true 2026-02-13T02:41:14.712 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_reported_epoch]=22 2026-02-13T02:41:14.712 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_seq]=60 2026-02-13T02:41:14.712 INFO:tasks.workunit.client.0.trial112.stdout:) 2026-02-13T02:41:14.722 INFO:tasks.workunit.client.0.trial112.stdout:test counter @ should be higher than before the unset: 2826243 2026-02-13T02:41:14.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:521: TEST_just_deep_scrubs: echo 'test counter @ should be higher than before the unset: ' 2826243 2026-02-13T02:41:14.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:523: TEST_just_deep_scrubs: expct_qry_duration=(['query_last_duration']='0' ['query_last_duration_neg']='not0') 2026-02-13T02:41:14.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:523: TEST_just_deep_scrubs: declare -A expct_qry_duration 2026-02-13T02:41:14.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:524: TEST_just_deep_scrubs: sc_data_2=() 2026-02-13T02:41:14.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:525: TEST_just_deep_scrubs: wait_any_cond 1.2 10 2026-02-13T02:40:45.260162+0000 expct_qry_duration 'WaitingAfterScrub ' sc_data_2 2026-02-13T02:41:14.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:106: wait_any_cond: local pgid=1.2 2026-02-13T02:41:14.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local retries=10 2026-02-13T02:41:14.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: local cmp_date=2026-02-13T02:40:45.260162+0000 2026-02-13T02:41:14.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:109: wait_any_cond: local -n ep=expct_qry_duration 2026-02-13T02:41:14.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:110: wait_any_cond: local -n out_array=sc_data_2 2026-02-13T02:41:14.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:111: wait_any_cond: local -A sc_data 2026-02-13T02:41:14.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:112: wait_any_cond: local extr_dbg=2 2026-02-13T02:41:14.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:115: wait_any_cond: local saved_echo_flag=x 2026-02-13T02:41:14.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:116: wait_any_cond: set +x 2026-02-13T02:41:14.725 INFO:tasks.workunit.client.0.trial112.stdout:waiting for any condition (WaitingAfterScrub ): pg:1.2 dt:2026-02-13T02:40:45.260162+0000 (10 retries) 2026-02-13T02:41:15.377 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:41:15.387 INFO:tasks.workunit.client.0.trial112.stdout:from pg dump pg: { 2026-02-13T02:41:15.387 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_pg_state": "active+clean", 2026-02-13T02:41:15.387 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_state_has_scrubbing": false, 2026-02-13T02:41:15.388 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_last_duration": 1, 2026-02-13T02:41:15.388 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_schedule": "periodic scrub scheduled", 2026-02-13T02:41:15.388 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_schedule_at": "2026-02-14T02:41:06.676139+0000", 2026-02-13T02:41:15.388 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_is_future": true, 2026-02-13T02:41:15.388 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_vs_date": true, 2026-02-13T02:41:15.388 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_reported_epoch": 22, 2026-02-13T02:41:15.388 INFO:tasks.workunit.client.0.trial112.stdout: "dmp_seq": 60 2026-02-13T02:41:15.388 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:41:15.388 INFO:tasks.workunit.client.0.trial112.stdout:query output: 2026-02-13T02:41:15.458 INFO:tasks.workunit.client.0.trial112.stdout: "scrubber": { 2026-02-13T02:41:15.459 INFO:tasks.workunit.client.0.trial112.stdout: "active": false, 2026-02-13T02:41:15.459 INFO:tasks.workunit.client.0.trial112.stdout: "must_scrub": false, 2026-02-13T02:41:15.459 INFO:tasks.workunit.client.0.trial112.stdout: "must_deep_scrub": false, 2026-02-13T02:41:15.459 INFO:tasks.workunit.client.0.trial112.stdout: "must_repair": false, 2026-02-13T02:41:15.459 INFO:tasks.workunit.client.0.trial112.stdout: "scrub_reg_stamp": "2026-02-14T02:41:06.676139+0000", 2026-02-13T02:41:15.459 INFO:tasks.workunit.client.0.trial112.stdout: "schedule": "scrub scheduled @ 2026-02-14T02:41:06.676 (2026-02-14T02:41:06.676)", 2026-02-13T02:41:15.459 INFO:tasks.workunit.client.0.trial112.stdout: "test_sequence": 2826243 2026-02-13T02:41:15.459 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:41:15.459 INFO:tasks.workunit.client.0.trial112.stdout: "agent_state": {} 2026-02-13T02:41:15.546 INFO:tasks.workunit.client.0.trial112.stdout:( 2026-02-13T02:41:15.546 INFO:tasks.workunit.client.0.trial112.stdout:[query_epoch]=22 2026-02-13T02:41:15.546 INFO:tasks.workunit.client.0.trial112.stdout:[query_seq]=61 2026-02-13T02:41:15.546 INFO:tasks.workunit.client.0.trial112.stdout:[query_active]=false 2026-02-13T02:41:15.546 INFO:tasks.workunit.client.0.trial112.stdout:[query_schedule]='scrub scheduled' 2026-02-13T02:41:15.546 INFO:tasks.workunit.client.0.trial112.stdout:[query_schedule_at]='2026-02-14T02:41:06.676' 2026-02-13T02:41:15.546 INFO:tasks.workunit.client.0.trial112.stdout:[query_target_at]='2026-02-14T02:41:06.676' 2026-02-13T02:41:15.546 INFO:tasks.workunit.client.0.trial112.stdout:[query_last_duration]=1 2026-02-13T02:41:15.547 INFO:tasks.workunit.client.0.trial112.stdout:[query_last_stamp]='2026-02-13T02:41:06.676139+0000' 2026-02-13T02:41:15.547 INFO:tasks.workunit.client.0.trial112.stdout:[query_last_scrub]='19x20' 2026-02-13T02:41:15.547 INFO:tasks.workunit.client.0.trial112.stdout:[query_is_future]=true 2026-02-13T02:41:15.547 INFO:tasks.workunit.client.0.trial112.stdout:[query_vs_date]=true 2026-02-13T02:41:15.547 INFO:tasks.workunit.client.0.trial112.stdout:[query_scrub_seq]=2826243 2026-02-13T02:41:15.547 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_pg_state]='active+clean' 2026-02-13T02:41:15.547 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_state_has_scrubbing]=false 2026-02-13T02:41:15.547 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_last_duration]=1 2026-02-13T02:41:15.547 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_schedule]='periodic scrub scheduled' 2026-02-13T02:41:15.547 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_schedule_at]='2026-02-14T02:41:06.676139+0000' 2026-02-13T02:41:15.547 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_is_future]=true 2026-02-13T02:41:15.548 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_vs_date]=true 2026-02-13T02:41:15.548 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_reported_epoch]=22 2026-02-13T02:41:15.548 INFO:tasks.workunit.client.0.trial112.stdout:[dmp_seq]=60 2026-02-13T02:41:15.548 INFO:tasks.workunit.client.0.trial112.stdout:) 2026-02-13T02:41:15.558 INFO:tasks.workunit.client.0.trial112.stdout:--> loop: 1 ~ false / 61 / 60 / true / 2026-02-13T02:41:06.676139+0000 / scrub scheduled %%% query_last_duration_neg query_last_duration 2026-02-13T02:41:15.559 INFO:tasks.workunit.client.0.trial112.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 1 2026-02-13T02:41:15.559 INFO:tasks.workunit.client.0.trial112.stdout:WaitingAfterScrub - 'query_last_duration' actual value (1) matches expected (0) (negation: 1) 2026-02-13T02:41:15.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:157: wait_any_cond: return 0 2026-02-13T02:41:15.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:526: TEST_just_deep_scrubs: perf_counters td/osd-scrub-test 3 2026-02-13T02:41:15.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2026-02-13T02:41:15.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=3 2026-02-13T02:41:15.560 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 3 - 1 2026-02-13T02:41:15.561 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 2 2026-02-13T02:41:15.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:41:15.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2026-02-13T02:41:15.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-13T02:41:15.640 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:41:15.641 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_ec": [ 2026-02-13T02:41:15.641 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:41:15.641 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:41:15.641 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:41:15.641 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:41:15.641 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:41:15.641 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:41:15.641 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:41:15.642 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:41:15.642 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:41:15.642 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:41:15.642 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:41:15.642 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:15.642 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:15.642 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:41:15.642 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_repl": [ 2026-02-13T02:41:15.642 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:41:15.642 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:41:15.642 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:41:15.643 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:41:15.643 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:41:15.643 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:41:15.643 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:41:15.643 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 2, 2026-02-13T02:41:15.643 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:41:15.643 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:41:15.643 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:41:15.643 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:15.643 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:15.643 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:41:15.644 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_ec": [ 2026-02-13T02:41:15.644 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:41:15.644 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:41:15.644 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:41:15.644 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:41:15.644 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:41:15.644 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:41:15.644 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:41:15.644 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:41:15.644 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:41:15.644 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:41:15.645 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:41:15.645 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:15.645 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:15.645 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:41:15.645 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_repl": [ 2026-02-13T02:41:15.645 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:41:15.645 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:41:15.645 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:41:15.645 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:41:15.645 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:41:15.645 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:41:15.646 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:41:15.646 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:41:15.646 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:41:15.646 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:41:15.646 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:41:15.646 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:15.646 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:15.646 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:41:15.646 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:41:15.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:41:15.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2026-02-13T02:41:15.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-13T02:41:15.710 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:41:15.710 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_ec": [ 2026-02-13T02:41:15.710 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:41:15.710 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:41:15.710 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:41:15.710 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:41:15.711 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:41:15.711 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:41:15.711 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:41:15.711 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:41:15.711 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:41:15.711 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:41:15.711 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:41:15.711 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:15.711 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:15.711 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:41:15.711 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_repl": [ 2026-02-13T02:41:15.712 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:41:15.712 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:41:15.712 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:41:15.712 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:41:15.712 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:41:15.712 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:41:15.712 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:41:15.712 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:41:15.712 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:41:15.712 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:41:15.712 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:41:15.712 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:15.713 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:15.713 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:41:15.713 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_ec": [ 2026-02-13T02:41:15.713 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:41:15.713 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:41:15.713 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:41:15.713 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:41:15.713 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:41:15.713 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:41:15.713 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:41:15.713 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:41:15.714 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:41:15.714 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:41:15.714 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:41:15.714 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:15.714 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:15.714 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:41:15.714 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_repl": [ 2026-02-13T02:41:15.714 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:41:15.714 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:41:15.714 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:41:15.714 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:41:15.715 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:41:15.715 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:41:15.715 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:41:15.715 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:41:15.715 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:41:15.715 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:41:15.715 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:41:15.715 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:15.715 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:15.715 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:41:15.715 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:41:15.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:41:15.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.2 counter dump 2026-02-13T02:41:15.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-13T02:41:15.781 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:41:15.781 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_ec": [ 2026-02-13T02:41:15.781 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:41:15.781 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:41:15.781 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:41:15.782 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:41:15.782 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:41:15.782 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:41:15.782 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:41:15.782 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:41:15.782 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:41:15.782 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:41:15.782 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:41:15.782 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:15.782 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:15.782 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:41:15.783 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_repl": [ 2026-02-13T02:41:15.783 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:41:15.783 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:41:15.783 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:41:15.783 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:41:15.783 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:41:15.783 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:41:15.783 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:41:15.783 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:41:15.783 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:41:15.783 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:41:15.783 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:41:15.784 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:15.784 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:15.784 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:41:15.784 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_ec": [ 2026-02-13T02:41:15.784 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:41:15.784 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:41:15.784 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:41:15.784 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:41:15.784 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:41:15.784 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:41:15.784 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:41:15.785 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:41:15.785 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:41:15.785 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:41:15.785 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:41:15.785 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:15.785 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:15.785 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:41:15.785 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_repl": [ 2026-02-13T02:41:15.785 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:41:15.785 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:41:15.785 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:41:15.786 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:41:15.786 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:41:15.786 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:41:15.786 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:41:15.786 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:41:15.786 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:41:15.786 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:41:15.786 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:41:15.786 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:15.786 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:15.786 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:41:15.787 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:41:15.787 INFO:tasks.workunit.client.0.trial112.stdout:-------------- Teardown Test TEST_just_deep_scrubs ------------------ 2026-02-13T02:41:15.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_just_deep_scrubs ------------------' 2026-02-13T02:41:15.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2026-02-13T02:41:15.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-13T02:41:15.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:41:15.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-13T02:41:15.789 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:41:15.789 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:41:15.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:41:15.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:41:15.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:41:15.892 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:41:15.893 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:41:15.893 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:41:15.893 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:41:15.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:41:15.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:41:15.895 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:41:15.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:41:15.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:41:15.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:41:15.897 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:41:15.898 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:41:15.899 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:41:15.899 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-13T02:41:15.914 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:41:15.914 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:41:15.914 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:41:15.914 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.508946 2026-02-13T02:41:15.915 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:41:15.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:41:15.916 INFO:tasks.workunit.client.0.trial112.stdout:-------------- Complete Test TEST_just_deep_scrubs ------------------ 2026-02-13T02:41:15.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_just_deep_scrubs ------------------' 2026-02-13T02:41:15.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2026-02-13T02:41:15.917 INFO:tasks.workunit.client.0.trial112.stdout:-------------- Prepare Test TEST_pg_dump_objects_scrubbed ------------------- 2026-02-13T02:41:15.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_pg_dump_objects_scrubbed -------------------' 2026-02-13T02:41:15.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2026-02-13T02:41:15.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2026-02-13T02:41:15.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2026-02-13T02:41:15.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-13T02:41:15.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:41:15.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-13T02:41:15.919 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:41:15.919 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:41:15.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:41:15.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:41:15.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:41:15.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:41:15.920 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:41:15.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:41:15.920 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:41:15.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:41:15.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:41:15.922 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:41:15.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:41:15.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:41:15.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:41:15.924 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:41:15.925 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:41:15.926 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:41:15.926 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-13T02:41:15.927 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:41:15.928 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:41:15.928 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:41:15.928 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.508946 2026-02-13T02:41:15.929 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:41:15.929 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:41:15.929 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2026-02-13T02:41:15.930 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T02:41:15.930 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:41:15.931 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:41:15.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.508946 2026-02-13T02:41:15.932 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T02:41:15.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-13T02:41:15.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T02:41:15.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2026-02-13T02:41:15.933 INFO:tasks.workunit.client.0.trial112.stdout:-------------- Run Test TEST_pg_dump_objects_scrubbed ----------------------- 2026-02-13T02:41:15.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_pg_dump_objects_scrubbed -----------------------' 2026-02-13T02:41:15.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_pg_dump_objects_scrubbed td/osd-scrub-test 2026-02-13T02:41:15.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:653: TEST_pg_dump_objects_scrubbed: local dir=td/osd-scrub-test 2026-02-13T02:41:15.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:654: TEST_pg_dump_objects_scrubbed: local poolname=test 2026-02-13T02:41:15.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:655: TEST_pg_dump_objects_scrubbed: local OSDS=3 2026-02-13T02:41:15.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:656: TEST_pg_dump_objects_scrubbed: local objects=15 2026-02-13T02:41:15.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:657: TEST_pg_dump_objects_scrubbed: local timeout=10 2026-02-13T02:41:15.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:659: TEST_pg_dump_objects_scrubbed: TESTDATA=testdata.508946 2026-02-13T02:41:15.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:661: TEST_pg_dump_objects_scrubbed: setup td/osd-scrub-test 2026-02-13T02:41:15.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2026-02-13T02:41:15.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2026-02-13T02:41:15.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-13T02:41:15.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:41:15.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-13T02:41:15.937 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:41:15.937 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:41:15.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:41:15.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:41:15.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:41:15.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:41:15.938 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:41:15.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:41:15.939 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:41:15.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:41:15.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:41:15.940 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:41:15.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:41:15.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:41:15.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:41:15.942 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:41:15.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:41:15.944 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:41:15.944 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-13T02:41:15.945 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:41:15.946 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:41:15.946 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:41:15.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.508946 2026-02-13T02:41:15.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:41:15.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:41:15.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2026-02-13T02:41:15.948 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T02:41:15.949 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:41:15.949 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:41:15.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.508946 2026-02-13T02:41:15.950 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T02:41:15.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-13T02:41:15.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T02:41:15.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2026-02-13T02:41:15.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:662: TEST_pg_dump_objects_scrubbed: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2026-02-13T02:41:15.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-test 2026-02-13T02:41:15.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-13T02:41:15.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-13T02:41:15.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-13T02:41:15.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-test/a 2026-02-13T02:41:15.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2026-02-13T02:41:15.973 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-13T02:41:15.973 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:41:15.973 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:41:15.973 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:41:15.973 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:41:15.973 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:41:15.974 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:41:15.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2026-02-13T02:41:16.001 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-13T02:41:16.002 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-13T02:41:16.002 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:41:16.002 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:41:16.002 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-13T02:41:16.002 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-13T02:41:16.004 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:41:16.004 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:41:16.004 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:41:16.004 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:41:16.004 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:41:16.004 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:41:16.004 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.508946/ceph-mon.a.asok 2026-02-13T02:41:16.004 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:41:16.004 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.508946/ceph-mon.a.asok config get fsid 2026-02-13T02:41:16.059 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-13T02:41:16.059 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:41:16.059 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:41:16.059 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-13T02:41:16.060 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-13T02:41:16.060 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:41:16.060 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:41:16.060 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:41:16.060 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:41:16.061 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:41:16.061 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:41:16.061 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.508946/ceph-mon.a.asok 2026-02-13T02:41:16.061 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:41:16.061 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.508946/ceph-mon.a.asok config get mon_host 2026-02-13T02:41:16.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:663: TEST_pg_dump_objects_scrubbed: run_mgr td/osd-scrub-test x --mgr_stats_period=1 2026-02-13T02:41:16.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-test 2026-02-13T02:41:16.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-13T02:41:16.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-13T02:41:16.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-13T02:41:16.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-test/x 2026-02-13T02:41:16.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-13T02:41:16.210 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-13T02:41:16.210 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:41:16.211 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:41:16.211 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:41:16.211 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:41:16.211 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:41:16.212 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:41:16.212 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:41:16.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mgr_stats_period=1 2026-02-13T02:41:16.230 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:664: TEST_pg_dump_objects_scrubbed: expr 3 - 1 2026-02-13T02:41:16.230 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:664: TEST_pg_dump_objects_scrubbed: seq 0 2 2026-02-13T02:41:16.233 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:664: TEST_pg_dump_objects_scrubbed: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:41:16.233 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:666: TEST_pg_dump_objects_scrubbed: run_osd td/osd-scrub-test 0 2026-02-13T02:41:16.233 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-13T02:41:16.233 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:41:16.233 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-13T02:41:16.233 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:41:16.233 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/0 2026-02-13T02:41:16.233 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-13T02:41:16.233 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:41:16.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:41:16.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:41:16.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2026-02-13T02:41:16.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2026-02-13T02:41:16.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:41:16.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:41:16.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-13T02:41:16.234 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:41:16.234 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:41:16.234 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:41:16.234 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:41:16.234 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:41:16.234 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:41:16.235 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:41:16.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:41:16.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:41:16.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:41:16.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:41:16.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-13T02:41:16.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-13T02:41:16.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:41:16.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:41:16.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:41:16.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:41:16.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:41:16.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:41:16.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/0 2026-02-13T02:41:16.237 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:41:16.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=9e77bec5-3a67-4b80-8bf4-845f1c3fc56e 2026-02-13T02:41:16.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 9e77bec5-3a67-4b80-8bf4-845f1c3fc56e' 2026-02-13T02:41:16.238 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 9e77bec5-3a67-4b80-8bf4-845f1c3fc56e 2026-02-13T02:41:16.238 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:41:16.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBMj45pVzZxDhAAx5+l34zZOD2qGEZsJGC6Xg== 2026-02-13T02:41:16.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBMj45pVzZxDhAAx5+l34zZOD2qGEZsJGC6Xg=="}' 2026-02-13T02:41:16.247 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 9e77bec5-3a67-4b80-8bf4-845f1c3fc56e -i td/osd-scrub-test/0/new.json 2026-02-13T02:41:16.337 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:41:16.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/0/new.json 2026-02-13T02:41:16.347 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBMj45pVzZxDhAAx5+l34zZOD2qGEZsJGC6Xg== --osd-uuid 9e77bec5-3a67-4b80-8bf4-845f1c3fc56e 2026-02-13T02:41:16.363 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:16.358+0000 7f280b256a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:41:16.364 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:16.358+0000 7f280b256a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:41:16.365 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:16.358+0000 7f280b256a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:41:16.365 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:16.358+0000 7f280b256a40 -1 bdev(0x55e52cae8c00 td/osd-scrub-test/0/block) open stat got: (1) Operation not permitted 2026-02-13T02:41:16.365 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:16.358+0000 7f280b256a40 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2026-02-13T02:41:17.042 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2026-02-13T02:41:17.042 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:41:17.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-13T02:41:17.043 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T02:41:17.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:41:17.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-13T02:41:17.145 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T02:41:17.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:41:17.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:41:17.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:41:17.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:41:17.167 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:17.162+0000 7f9e84a6ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:41:17.169 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:17.162+0000 7f9e84a6ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:41:17.171 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:17.166+0000 7f9e84a6ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:41:17.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-13T02:41:17.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:41:17.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:41:17.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:41:17.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:41:17.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:41:17.238 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:41:17.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:41:17.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:41:17.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:41:17.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:41:17.676 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:17.670+0000 7f9e84a6ba40 -1 Falling back to public interface 2026-02-13T02:41:18.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:41:18.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:41:18.330 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:41:18.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:41:18.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:41:18.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:41:18.441 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:41:18.664 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:18.658+0000 7f9e84a6ba40 -1 osd.0 0 log_to_monitors true 2026-02-13T02:41:19.442 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:41:19.442 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:41:19.443 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:41:19.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:41:19.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:41:19.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:41:19.601 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1404280334,v1:127.0.0.1:6801/1404280334] [v2:127.0.0.1:6802/1404280334,v1:127.0.0.1:6803/1404280334] exists,up 9e77bec5-3a67-4b80-8bf4-845f1c3fc56e 2026-02-13T02:41:19.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:41:19.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:41:19.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:41:19.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:664: TEST_pg_dump_objects_scrubbed: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:41:19.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:666: TEST_pg_dump_objects_scrubbed: run_osd td/osd-scrub-test 1 2026-02-13T02:41:19.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-13T02:41:19.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:41:19.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-13T02:41:19.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:41:19.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/1 2026-02-13T02:41:19.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-13T02:41:19.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:41:19.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:41:19.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:41:19.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2026-02-13T02:41:19.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2026-02-13T02:41:19.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:41:19.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:41:19.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-13T02:41:19.606 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:41:19.606 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:41:19.606 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:41:19.606 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:41:19.606 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:41:19.606 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:41:19.606 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:41:19.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:41:19.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:41:19.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:41:19.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:41:19.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-13T02:41:19.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-13T02:41:19.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:41:19.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:41:19.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:41:19.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:41:19.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:41:19.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:41:19.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/1 2026-02-13T02:41:19.608 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:41:19.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=061f94dc-7140-4a04-b2be-5736e2f2f1b7 2026-02-13T02:41:19.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 061f94dc-7140-4a04-b2be-5736e2f2f1b7' 2026-02-13T02:41:19.609 INFO:tasks.workunit.client.0.trial112.stdout:add osd1 061f94dc-7140-4a04-b2be-5736e2f2f1b7 2026-02-13T02:41:19.609 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:41:19.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBPj45psauFJBAAxmz5FRY2NWbL5v43+poi2g== 2026-02-13T02:41:19.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBPj45psauFJBAAxmz5FRY2NWbL5v43+poi2g=="}' 2026-02-13T02:41:19.618 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 061f94dc-7140-4a04-b2be-5736e2f2f1b7 -i td/osd-scrub-test/1/new.json 2026-02-13T02:41:19.772 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:41:19.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/1/new.json 2026-02-13T02:41:19.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBPj45psauFJBAAxmz5FRY2NWbL5v43+poi2g== --osd-uuid 061f94dc-7140-4a04-b2be-5736e2f2f1b7 2026-02-13T02:41:19.808 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:19.802+0000 7fd58d575a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:41:19.810 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:19.802+0000 7fd58d575a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:41:19.812 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:19.806+0000 7fd58d575a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:41:19.812 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:19.806+0000 7fd58d575a40 -1 bdev(0x556862900c00 td/osd-scrub-test/1/block) open stat got: (1) Operation not permitted 2026-02-13T02:41:19.813 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:19.806+0000 7fd58d575a40 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2026-02-13T02:41:20.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2026-02-13T02:41:20.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:41:20.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-13T02:41:20.452 INFO:tasks.workunit.client.0.trial112.stdout:adding osd1 key to auth repository 2026-02-13T02:41:20.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:41:20.655 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T02:41:20.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-13T02:41:20.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:41:20.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:41:20.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:41:20.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:41:20.677 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:20.670+0000 7f88f3898a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:41:20.678 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:20.674+0000 7f88f3898a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:41:20.679 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:20.674+0000 7f88f3898a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:41:20.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-13T02:41:20.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:41:20.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:41:20.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:41:20.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:41:20.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:41:20.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:41:20.820 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:41:20.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:41:20.821 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:41:20.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:41:21.076 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:21.070+0000 7f88f3898a40 -1 Falling back to public interface 2026-02-13T02:41:21.725 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:21.718+0000 7f88f3898a40 -1 osd.1 0 log_to_monitors true 2026-02-13T02:41:21.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:41:21.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:41:21.987 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:41:21.988 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:41:21.988 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:41:21.988 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:41:22.154 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:41:23.155 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:41:23.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:41:23.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:41:23.156 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:41:23.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:41:23.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:41:23.312 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/457263124,v1:127.0.0.1:6811/457263124] [v2:127.0.0.1:6812/457263124,v1:127.0.0.1:6813/457263124] exists,up 061f94dc-7140-4a04-b2be-5736e2f2f1b7 2026-02-13T02:41:23.315 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:41:23.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:41:23.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:41:23.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:664: TEST_pg_dump_objects_scrubbed: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:41:23.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:666: TEST_pg_dump_objects_scrubbed: run_osd td/osd-scrub-test 2 2026-02-13T02:41:23.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-13T02:41:23.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:41:23.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-13T02:41:23.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:41:23.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/2 2026-02-13T02:41:23.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-13T02:41:23.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:41:23.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:41:23.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:41:23.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2026-02-13T02:41:23.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2026-02-13T02:41:23.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:41:23.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:41:23.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-13T02:41:23.317 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:41:23.317 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:41:23.317 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:41:23.317 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:41:23.317 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:41:23.317 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:41:23.318 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:41:23.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:41:23.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:41:23.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:41:23.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:41:23.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-13T02:41:23.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-13T02:41:23.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:41:23.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:41:23.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:41:23.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:41:23.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:41:23.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:41:23.319 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/2 2026-02-13T02:41:23.319 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:41:23.319 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=72ce02b1-ef5c-4d86-b86c-a263bfa1f4b1 2026-02-13T02:41:23.320 INFO:tasks.workunit.client.0.trial112.stdout:add osd2 72ce02b1-ef5c-4d86-b86c-a263bfa1f4b1 2026-02-13T02:41:23.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 72ce02b1-ef5c-4d86-b86c-a263bfa1f4b1' 2026-02-13T02:41:23.320 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:41:23.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBTj45pTbhWExAAVMCLZu8i6rl4RRQQWB9aBQ== 2026-02-13T02:41:23.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBTj45pTbhWExAAVMCLZu8i6rl4RRQQWB9aBQ=="}' 2026-02-13T02:41:23.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 72ce02b1-ef5c-4d86-b86c-a263bfa1f4b1 -i td/osd-scrub-test/2/new.json 2026-02-13T02:41:23.492 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:41:23.502 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/2/new.json 2026-02-13T02:41:23.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBTj45pTbhWExAAVMCLZu8i6rl4RRQQWB9aBQ== --osd-uuid 72ce02b1-ef5c-4d86-b86c-a263bfa1f4b1 2026-02-13T02:41:23.519 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:23.514+0000 7f240e313a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:41:23.520 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:23.514+0000 7f240e313a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:41:23.521 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:23.514+0000 7f240e313a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:41:23.521 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:23.514+0000 7f240e313a40 -1 bdev(0x55a502570c00 td/osd-scrub-test/2/block) open stat got: (1) Operation not permitted 2026-02-13T02:41:23.522 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:23.514+0000 7f240e313a40 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2026-02-13T02:41:24.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2026-02-13T02:41:24.147 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:41:24.147 INFO:tasks.workunit.client.0.trial112.stdout:adding osd2 key to auth repository 2026-02-13T02:41:24.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-13T02:41:24.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:41:24.352 INFO:tasks.workunit.client.0.trial112.stdout:start osd.2 2026-02-13T02:41:24.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-13T02:41:24.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:41:24.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:41:24.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:41:24.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:41:24.374 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:24.366+0000 7f1912aeca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:41:24.375 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:24.370+0000 7f1912aeca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:41:24.377 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:24.370+0000 7f1912aeca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:41:24.513 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-13T02:41:24.513 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:41:24.513 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T02:41:24.513 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:41:24.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:41:24.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:41:24.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:41:24.514 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:41:24.515 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:41:24.515 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:41:24.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:41:24.764 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:24.758+0000 7f1912aeca40 -1 Falling back to public interface 2026-02-13T02:41:25.340 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:25.334+0000 7f1912aeca40 -1 osd.2 0 log_to_monitors true 2026-02-13T02:41:25.678 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:41:25.678 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:41:25.678 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:41:25.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:41:25.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:41:25.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:41:25.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:41:26.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:41:26.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:41:26.863 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:41:26.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:41:26.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:41:26.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:41:27.019 INFO:tasks.workunit.client.0.trial112.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3177308625,v1:127.0.0.1:6819/3177308625] [v2:127.0.0.1:6820/3177308625,v1:127.0.0.1:6821/3177308625] exists,up 72ce02b1-ef5c-4d86-b86c-a263bfa1f4b1 2026-02-13T02:41:27.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:41:27.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:41:27.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:41:27.021 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:670: TEST_pg_dump_objects_scrubbed: create_pool test 1 1 2026-02-13T02:41:27.021 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2026-02-13T02:41:27.226 INFO:tasks.workunit.client.0.trial112.stderr:pool 'test' created 2026-02-13T02:41:27.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:41:28.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:671: TEST_pg_dump_objects_scrubbed: wait_for_clean 2026-02-13T02:41:28.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:41:28.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:41:28.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:41:28.240 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:41:28.241 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:41:28.241 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:41:28.242 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:41:28.242 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:41:28.242 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:41:28.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:41:28.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:41:28.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:41:28.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:41:28.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:41:28.350 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:41:28.511 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:41:28.512 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:41:28.512 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:41:28.512 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:41:28.512 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:41:28.512 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:41:28.584 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836491 2026-02-13T02:41:28.584 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836491 2026-02-13T02:41:28.584 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491' 2026-02-13T02:41:28.584 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:41:28.585 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:41:28.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705671 2026-02-13T02:41:28.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705671 2026-02-13T02:41:28.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705671' 2026-02-13T02:41:28.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:41:28.660 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:41:28.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574852 2026-02-13T02:41:28.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574852 2026-02-13T02:41:28.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705671 2-55834574852' 2026-02-13T02:41:28.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:41:28.733 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836491 2026-02-13T02:41:28.733 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:41:28.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:41:28.735 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836491 2026-02-13T02:41:28.735 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:41:28.737 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836491 2026-02-13T02:41:28.737 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836491 2026-02-13T02:41:28.737 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2026-02-13T02:41:28.737 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:41:28.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836490 -lt 21474836491 2026-02-13T02:41:28.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:41:29.904 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:41:29.905 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:41:30.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836490 -lt 21474836491 2026-02-13T02:41:30.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:41:31.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 499 -eq 0 ']' 2026-02-13T02:41:31.070 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:41:31.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836493 -lt 21474836491 2026-02-13T02:41:31.228 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:41:31.228 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705671 2026-02-13T02:41:31.228 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:41:31.230 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:41:31.230 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705671 2026-02-13T02:41:31.231 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:41:31.232 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705671 2026-02-13T02:41:31.232 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705671' 2026-02-13T02:41:31.233 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705671 2026-02-13T02:41:31.233 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:41:31.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705673 -lt 38654705671 2026-02-13T02:41:31.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:41:31.397 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574852 2026-02-13T02:41:31.397 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:41:31.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:41:31.399 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574852 2026-02-13T02:41:31.399 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:41:31.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574852 2026-02-13T02:41:31.400 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574852 2026-02-13T02:41:31.401 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574852' 2026-02-13T02:41:31.401 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:41:31.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574853 -lt 55834574852 2026-02-13T02:41:31.564 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:41:31.565 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:41:31.565 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:41:31.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:41:31.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:41:31.755 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:41:31.755 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:41:31.755 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:41:31.755 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:41:31.755 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:41:31.755 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:41:31.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:41:31.910 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:41:31.910 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:41:31.910 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:41:32.099 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:41:32.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:41:32.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:41:32.100 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:672: TEST_pg_dump_objects_scrubbed: ceph osd dump 2026-02-13T02:41:32.100 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:672: TEST_pg_dump_objects_scrubbed: grep '^pool.*['\'']test['\'']' 2026-02-13T02:41:32.101 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:672: TEST_pg_dump_objects_scrubbed: awk '{ print $2 }' 2026-02-13T02:41:32.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:672: TEST_pg_dump_objects_scrubbed: poolid=1 2026-02-13T02:41:32.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:674: TEST_pg_dump_objects_scrubbed: dd if=/dev/urandom of=testdata.508946 bs=1032 count=1 2026-02-13T02:41:32.260 INFO:tasks.workunit.client.0.trial112.stderr:1+0 records in 2026-02-13T02:41:32.261 INFO:tasks.workunit.client.0.trial112.stderr:1+0 records out 2026-02-13T02:41:32.261 INFO:tasks.workunit.client.0.trial112.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 5.359e-05 s, 19.3 MB/s 2026-02-13T02:41:32.261 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: seq 1 15 2026-02-13T02:41:32.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-13T02:41:32.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj1 testdata.508946 2026-02-13T02:41:32.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-13T02:41:32.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj2 testdata.508946 2026-02-13T02:41:32.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-13T02:41:32.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj3 testdata.508946 2026-02-13T02:41:32.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-13T02:41:32.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj4 testdata.508946 2026-02-13T02:41:32.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-13T02:41:32.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj5 testdata.508946 2026-02-13T02:41:32.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-13T02:41:32.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj6 testdata.508946 2026-02-13T02:41:32.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-13T02:41:32.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj7 testdata.508946 2026-02-13T02:41:32.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-13T02:41:32.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj8 testdata.508946 2026-02-13T02:41:32.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-13T02:41:32.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj9 testdata.508946 2026-02-13T02:41:32.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-13T02:41:32.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj10 testdata.508946 2026-02-13T02:41:32.497 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-13T02:41:32.497 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj11 testdata.508946 2026-02-13T02:41:32.515 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-13T02:41:32.515 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj12 testdata.508946 2026-02-13T02:41:32.537 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-13T02:41:32.537 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj13 testdata.508946 2026-02-13T02:41:32.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-13T02:41:32.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj14 testdata.508946 2026-02-13T02:41:32.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-13T02:41:32.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj15 testdata.508946 2026-02-13T02:41:32.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:679: TEST_pg_dump_objects_scrubbed: rm -f testdata.508946 2026-02-13T02:41:32.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:681: TEST_pg_dump_objects_scrubbed: local pgid=1.0 2026-02-13T02:41:32.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:683: TEST_pg_dump_objects_scrubbed: pg_scrub 1.0 2026-02-13T02:41:32.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=1.0 2026-02-13T02:41:32.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 1.0 2026-02-13T02:41:32.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=1.0 2026-02-13T02:41:32.606 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-13T02:41:32.607 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:41:32.607 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:41:32.607 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:41:32.607 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:41:32.607 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:41:32.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-13T02:41:32.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-13T02:41:32.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-13T02:41:32.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-13T02:41:32.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:41:32.897 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:41:33.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:41:33.062 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:41:33.062 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:41:33.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:41:33.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:41:33.063 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:41:33.132 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836497 2026-02-13T02:41:33.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836497 2026-02-13T02:41:33.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497' 2026-02-13T02:41:33.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:41:33.133 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:41:33.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705677 2026-02-13T02:41:33.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705677 2026-02-13T02:41:33.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-38654705677' 2026-02-13T02:41:33.205 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:41:33.205 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:41:33.277 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574857 2026-02-13T02:41:33.277 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574857 2026-02-13T02:41:33.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-38654705677 2-55834574857' 2026-02-13T02:41:33.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:41:33.278 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836497 2026-02-13T02:41:33.278 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:41:33.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:41:33.280 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836497 2026-02-13T02:41:33.280 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:41:33.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836497 2026-02-13T02:41:33.282 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836497 2026-02-13T02:41:33.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2026-02-13T02:41:33.282 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:41:33.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836495 -lt 21474836497 2026-02-13T02:41:33.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:41:34.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:41:34.447 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:41:34.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836498 -lt 21474836497 2026-02-13T02:41:34.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:41:34.614 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705677 2026-02-13T02:41:34.614 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:41:34.615 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:41:34.616 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705677 2026-02-13T02:41:34.616 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:41:34.618 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705677 2026-02-13T02:41:34.618 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705677' 2026-02-13T02:41:34.619 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705677 2026-02-13T02:41:34.619 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:41:34.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705678 -lt 38654705677 2026-02-13T02:41:34.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:41:34.789 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574857 2026-02-13T02:41:34.790 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:41:34.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:41:34.791 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574857 2026-02-13T02:41:34.792 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:41:34.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574857 2026-02-13T02:41:34.793 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574857 2026-02-13T02:41:34.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574857' 2026-02-13T02:41:34.793 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:41:34.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574858 -lt 55834574857 2026-02-13T02:41:34.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-13T02:41:34.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 0' 2026-02-13T02:41:34.965 INFO:tasks.workunit.client.0.trial112.stdout:#---------- 1.0 loop 0 2026-02-13T02:41:34.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-13T02:41:34.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-13T02:41:34.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-13T02:41:34.966 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-13T02:41:34.966 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-13T02:41:35.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-13T02:41:35.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-13T02:41:35.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-13T02:41:35.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-13T02:41:35.042 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: get_last_scrub_stamp 1.0 2026-02-13T02:41:35.042 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:41:35.042 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:41:35.042 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:41:35.042 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:41:35.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: local last_scrub=2026-02-13T02:41:27.220950+0000 2026-02-13T02:41:35.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_scrub: ceph pg scrub 1.0 2026-02-13T02:41:35.344 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 1.0 on osd.1 to scrub 2026-02-13T02:41:35.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_scrub: wait_for_scrub 1.0 2026-02-13T02:41:27.220950+0000 2026-02-13T02:41:35.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-13T02:41:35.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:41:27.220950+0000 2026-02-13T02:41:35.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:41:35.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:41:35.355 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:41:35.355 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:41:35.355 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:41:35.355 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:41:35.355 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:41:35.356 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:41:35.516 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:41:27.220950+0000 '>' 2026-02-13T02:41:27.220950+0000 2026-02-13T02:41:35.516 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:41:36.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:41:36.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:41:36.518 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:41:36.518 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:41:36.518 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:41:36.519 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:41:36.519 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:41:36.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:41:35.834883+0000 '>' 2026-02-13T02:41:27.220950+0000 2026-02-13T02:41:36.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:41:36.676 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:684: TEST_pg_dump_objects_scrubbed: ceph pg 1.0 query 2026-02-13T02:41:36.676 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:684: TEST_pg_dump_objects_scrubbed: jq .info.stats.objects_scrubbed 2026-02-13T02:41:36.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:684: TEST_pg_dump_objects_scrubbed: test 15 = 15 2026-02-13T02:41:36.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:685: TEST_pg_dump_objects_scrubbed: perf_counters td/osd-scrub-test 3 2026-02-13T02:41:36.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2026-02-13T02:41:36.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=3 2026-02-13T02:41:36.752 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 3 - 1 2026-02-13T02:41:36.753 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 2 2026-02-13T02:41:36.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:41:36.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2026-02-13T02:41:36.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-13T02:41:36.833 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:41:36.833 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_ec": [ 2026-02-13T02:41:36.834 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:41:36.834 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:41:36.834 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:41:36.834 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:41:36.834 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:41:36.834 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:41:36.834 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:41:36.834 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:41:36.834 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:41:36.834 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:41:36.834 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:41:36.834 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:36.835 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:36.835 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:41:36.835 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_repl": [ 2026-02-13T02:41:36.835 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:41:36.835 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:41:36.835 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:41:36.835 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:41:36.835 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:41:36.835 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:41:36.835 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:41:36.835 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:41:36.835 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:41:36.836 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:41:36.836 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:41:36.836 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:36.836 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:36.836 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:41:36.836 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_ec": [ 2026-02-13T02:41:36.836 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:41:36.836 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:41:36.836 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:41:36.836 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:41:36.836 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:41:36.836 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:41:36.836 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:41:36.837 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:41:36.837 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:41:36.837 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:41:36.837 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:41:36.837 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:36.837 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:36.837 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:41:36.837 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_repl": [ 2026-02-13T02:41:36.837 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:41:36.837 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:41:36.837 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:41:36.837 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:41:36.838 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:41:36.838 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:41:36.838 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:41:36.838 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:41:36.838 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:41:36.838 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:41:36.838 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:41:36.838 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:36.838 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:36.838 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:41:36.838 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:41:36.839 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:41:36.839 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2026-02-13T02:41:36.839 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-13T02:41:36.905 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:41:36.905 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_ec": [ 2026-02-13T02:41:36.905 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:41:36.905 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:41:36.905 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:41:36.905 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:41:36.906 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:41:36.906 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:41:36.906 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:41:36.906 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:41:36.906 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:41:36.906 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:41:36.906 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:41:36.906 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:36.906 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:36.906 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:41:36.906 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_repl": [ 2026-02-13T02:41:36.906 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:41:36.906 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:41:36.907 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:41:36.907 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:41:36.907 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:41:36.907 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:41:36.907 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:41:36.907 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:41:36.907 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:41:36.907 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:41:36.907 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:41:36.907 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:36.907 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:36.907 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:41:36.907 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_ec": [ 2026-02-13T02:41:36.908 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:41:36.908 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:41:36.908 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:41:36.908 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:41:36.908 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:41:36.908 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:41:36.908 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:41:36.908 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:41:36.908 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:41:36.908 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:41:36.908 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:41:36.908 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:36.908 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:36.908 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:41:36.909 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_repl": [ 2026-02-13T02:41:36.909 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:41:36.909 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:41:36.909 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:41:36.909 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:41:36.909 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:41:36.909 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:41:36.909 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:41:36.909 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 1, 2026-02-13T02:41:36.909 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:41:36.909 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:41:36.909 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:41:36.909 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:36.910 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:36.910 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:41:36.910 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:41:36.910 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:41:36.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.2 counter dump 2026-02-13T02:41:36.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-13T02:41:36.976 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:41:36.976 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_ec": [ 2026-02-13T02:41:36.976 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:41:36.976 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:41:36.976 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:41:36.976 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:41:36.977 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:41:36.977 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:41:36.977 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:41:36.977 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:41:36.977 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:41:36.977 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:41:36.977 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:41:36.977 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:36.977 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:36.977 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:41:36.977 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_repl": [ 2026-02-13T02:41:36.977 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:41:36.977 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:41:36.978 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:41:36.978 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:41:36.978 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:41:36.978 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:41:36.978 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:41:36.978 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:41:36.978 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:41:36.978 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:41:36.978 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:41:36.978 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:36.978 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:36.978 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:41:36.978 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_ec": [ 2026-02-13T02:41:36.979 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:41:36.979 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:41:36.979 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:41:36.979 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:41:36.979 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:41:36.979 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:41:36.979 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:41:36.979 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:41:36.979 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:41:36.979 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:41:36.979 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:41:36.979 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:36.979 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:36.980 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:41:36.980 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_repl": [ 2026-02-13T02:41:36.980 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:41:36.980 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:41:36.980 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:41:36.980 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:41:36.980 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:41:36.980 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:41:36.980 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:41:36.980 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:41:36.980 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:41:36.980 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:41:36.980 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:41:36.981 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:36.981 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:41:36.981 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:41:36.981 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:41:36.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:687: TEST_pg_dump_objects_scrubbed: teardown td/osd-scrub-test 2026-02-13T02:41:36.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-13T02:41:36.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:41:36.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-13T02:41:36.982 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:41:36.982 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:41:36.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:41:36.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:41:36.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:41:37.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:41:37.088 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:41:37.089 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:41:37.089 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:41:37.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:41:37.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:41:37.091 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:41:37.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:41:37.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:41:37.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:41:37.093 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:41:37.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:41:37.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:41:37.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-13T02:41:37.105 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:41:37.105 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:41:37.105 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:41:37.105 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.508946 2026-02-13T02:41:37.107 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:41:37.107 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:41:37.107 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_pg_dump_objects_scrubbed ------------------' 2026-02-13T02:41:37.107 INFO:tasks.workunit.client.0.trial112.stdout:-------------- Teardown Test TEST_pg_dump_objects_scrubbed ------------------ 2026-02-13T02:41:37.108 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2026-02-13T02:41:37.108 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-13T02:41:37.108 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:41:37.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-13T02:41:37.109 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:41:37.109 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:41:37.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:41:37.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:41:37.109 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:41:37.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:41:37.111 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:41:37.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:41:37.112 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:41:37.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:41:37.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:41:37.113 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:41:37.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:41:37.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:41:37.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:41:37.115 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:41:37.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:41:37.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:41:37.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-13T02:41:37.119 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:41:37.119 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:41:37.119 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:41:37.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.508946 2026-02-13T02:41:37.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:41:37.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:41:37.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_pg_dump_objects_scrubbed ------------------' 2026-02-13T02:41:37.121 INFO:tasks.workunit.client.0.trial112.stdout:-------------- Complete Test TEST_pg_dump_objects_scrubbed ------------------ 2026-02-13T02:41:37.121 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2026-02-13T02:41:37.121 INFO:tasks.workunit.client.0.trial112.stdout:-------------- Prepare Test TEST_scrub_abort ------------------- 2026-02-13T02:41:37.123 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_scrub_abort -------------------' 2026-02-13T02:41:37.123 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2026-02-13T02:41:37.123 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2026-02-13T02:41:37.123 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2026-02-13T02:41:37.123 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-13T02:41:37.123 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:41:37.123 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-13T02:41:37.123 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:41:37.123 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:41:37.123 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:41:37.123 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:41:37.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:41:37.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:41:37.124 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:41:37.125 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:41:37.125 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:41:37.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:41:37.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:41:37.127 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:41:37.128 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:41:37.128 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:41:37.129 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:41:37.129 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:41:37.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:41:37.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:41:37.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-13T02:41:37.133 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:41:37.133 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:41:37.133 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:41:37.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.508946 2026-02-13T02:41:37.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:41:37.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:41:37.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2026-02-13T02:41:37.136 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T02:41:37.136 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:41:37.136 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:41:37.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.508946 2026-02-13T02:41:37.138 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T02:41:37.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-13T02:41:37.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T02:41:37.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2026-02-13T02:41:37.139 INFO:tasks.workunit.client.0.trial112.stdout:-------------- Run Test TEST_scrub_abort ----------------------- 2026-02-13T02:41:37.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_scrub_abort -----------------------' 2026-02-13T02:41:37.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_scrub_abort td/osd-scrub-test 2026-02-13T02:41:37.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:404: TEST_scrub_abort: local dir=td/osd-scrub-test 2026-02-13T02:41:37.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:405: TEST_scrub_abort: _scrub_abort td/osd-scrub-test scrub 2026-02-13T02:41:37.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:293: _scrub_abort: local dir=td/osd-scrub-test 2026-02-13T02:41:37.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:294: _scrub_abort: local poolname=test 2026-02-13T02:41:37.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:295: _scrub_abort: local OSDS=3 2026-02-13T02:41:37.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:296: _scrub_abort: local objects=1000 2026-02-13T02:41:37.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:297: _scrub_abort: local type=scrub 2026-02-13T02:41:37.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:299: _scrub_abort: TESTDATA=testdata.508946 2026-02-13T02:41:37.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:300: _scrub_abort: test scrub = scrub 2026-02-13T02:41:37.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:302: _scrub_abort: stopscrub=noscrub 2026-02-13T02:41:37.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:303: _scrub_abort: check=noscrub 2026-02-13T02:41:37.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:309: _scrub_abort: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2026-02-13T02:41:37.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-test 2026-02-13T02:41:37.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-13T02:41:37.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-13T02:41:37.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-13T02:41:37.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-test/a 2026-02-13T02:41:37.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2026-02-13T02:41:37.161 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-13T02:41:37.161 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:41:37.161 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:41:37.162 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:41:37.162 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:41:37.162 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:41:37.162 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:41:37.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2026-02-13T02:41:37.190 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-13T02:41:37.191 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-13T02:41:37.191 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:41:37.191 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:41:37.191 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-13T02:41:37.192 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-13T02:41:37.192 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:41:37.192 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:41:37.192 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:41:37.193 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:41:37.193 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:41:37.193 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:41:37.193 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.508946/ceph-mon.a.asok 2026-02-13T02:41:37.193 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:41:37.193 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.508946/ceph-mon.a.asok config get fsid 2026-02-13T02:41:37.248 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-13T02:41:37.248 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:41:37.249 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:41:37.249 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-13T02:41:37.250 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-13T02:41:37.250 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:41:37.250 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:41:37.250 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:41:37.250 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:41:37.250 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:41:37.250 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:41:37.250 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.508946/ceph-mon.a.asok 2026-02-13T02:41:37.251 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:41:37.251 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.508946/ceph-mon.a.asok config get mon_host 2026-02-13T02:41:37.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:310: _scrub_abort: run_mgr td/osd-scrub-test x --mgr_stats_period=1 2026-02-13T02:41:37.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-test 2026-02-13T02:41:37.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-13T02:41:37.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-13T02:41:37.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-13T02:41:37.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-test/x 2026-02-13T02:41:37.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-13T02:41:37.400 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-13T02:41:37.400 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:41:37.400 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:41:37.401 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:41:37.401 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:41:37.401 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:41:37.401 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:41:37.402 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:41:37.403 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mgr_stats_period=1 2026-02-13T02:41:37.422 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:311: _scrub_abort: expr 3 - 1 2026-02-13T02:41:37.423 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:311: _scrub_abort: seq 0 2 2026-02-13T02:41:37.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:311: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:41:37.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:317: _scrub_abort: run_osd td/osd-scrub-test 0 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2026-02-13T02:41:37.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-13T02:41:37.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:41:37.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-13T02:41:37.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:41:37.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/0 2026-02-13T02:41:37.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-13T02:41:37.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:41:37.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:41:37.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:41:37.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2026-02-13T02:41:37.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2026-02-13T02:41:37.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:41:37.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:41:37.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-13T02:41:37.428 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:41:37.428 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:41:37.428 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:41:37.428 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:41:37.428 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:41:37.428 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:41:37.429 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:41:37.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:41:37.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:41:37.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:41:37.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:41:37.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-13T02:41:37.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-13T02:41:37.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:41:37.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:41:37.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:41:37.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:41:37.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:41:37.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2026-02-13T02:41:37.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/0 2026-02-13T02:41:37.430 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:41:37.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=32a2140e-7007-48df-9f42-601b77fae02f 2026-02-13T02:41:37.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 32a2140e-7007-48df-9f42-601b77fae02f' 2026-02-13T02:41:37.431 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 32a2140e-7007-48df-9f42-601b77fae02f 2026-02-13T02:41:37.431 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:41:37.439 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBhj45pjGPlGRAAiSniihHX6tCJyrRShiGiKA== 2026-02-13T02:41:37.439 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBhj45pjGPlGRAAiSniihHX6tCJyrRShiGiKA=="}' 2026-02-13T02:41:37.439 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 32a2140e-7007-48df-9f42-601b77fae02f -i td/osd-scrub-test/0/new.json 2026-02-13T02:41:37.527 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:41:37.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/0/new.json 2026-02-13T02:41:37.537 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQBhj45pjGPlGRAAiSniihHX6tCJyrRShiGiKA== --osd-uuid 32a2140e-7007-48df-9f42-601b77fae02f 2026-02-13T02:41:37.552 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:37.546+0000 7f05584aea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:41:37.553 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:37.546+0000 7f05584aea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:41:37.554 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:37.550+0000 7f05584aea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:41:37.555 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:37.550+0000 7f05584aea40 -1 bdev(0x55d1abbe8c00 td/osd-scrub-test/0/block) open stat got: (1) Operation not permitted 2026-02-13T02:41:37.555 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:37.550+0000 7f05584aea40 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2026-02-13T02:41:38.266 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2026-02-13T02:41:38.266 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:41:38.267 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-13T02:41:38.267 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T02:41:38.267 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:41:38.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-13T02:41:38.362 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T02:41:38.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2026-02-13T02:41:38.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:41:38.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:41:38.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:41:38.377 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:38.370+0000 7fca94e74a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:41:38.379 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:38.374+0000 7fca94e74a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:41:38.380 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:38.374+0000 7fca94e74a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:41:38.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-13T02:41:38.458 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:41:38.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:41:38.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:41:38.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:41:38.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:41:38.459 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:41:38.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:41:38.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:41:38.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:41:38.547 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:41:38.764 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:38.758+0000 7fca94e74a40 -1 Falling back to public interface 2026-02-13T02:41:39.396 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:39.390+0000 7fca94e74a40 -1 osd.0 0 log_to_monitors true 2026-02-13T02:41:39.548 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:41:39.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:41:39.549 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:41:39.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:41:39.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:41:39.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:41:39.663 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/44677598,v1:127.0.0.1:6801/44677598] [v2:127.0.0.1:6802/44677598,v1:127.0.0.1:6803/44677598] exists,up 32a2140e-7007-48df-9f42-601b77fae02f 2026-02-13T02:41:39.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:41:39.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:41:39.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:41:39.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:311: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:41:39.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:317: _scrub_abort: run_osd td/osd-scrub-test 1 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2026-02-13T02:41:39.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-13T02:41:39.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:41:39.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-13T02:41:39.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:41:39.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/1 2026-02-13T02:41:39.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-13T02:41:39.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:41:39.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:41:39.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:41:39.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2026-02-13T02:41:39.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2026-02-13T02:41:39.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:41:39.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:41:39.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-13T02:41:39.669 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:41:39.669 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:41:39.669 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:41:39.669 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:41:39.669 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:41:39.669 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:41:39.669 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:41:39.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:41:39.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:41:39.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:41:39.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:41:39.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-13T02:41:39.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-13T02:41:39.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:41:39.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:41:39.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:41:39.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:41:39.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:41:39.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2026-02-13T02:41:39.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/1 2026-02-13T02:41:39.670 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:41:39.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e5f0f6d4-6c65-44e3-97a9-6a92a2c2ad87 2026-02-13T02:41:39.671 INFO:tasks.workunit.client.0.trial112.stdout:add osd1 e5f0f6d4-6c65-44e3-97a9-6a92a2c2ad87 2026-02-13T02:41:39.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 e5f0f6d4-6c65-44e3-97a9-6a92a2c2ad87' 2026-02-13T02:41:39.671 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:41:39.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBjj45pkFYVKBAAMDj/LwBaJ0oIt2mWzGKaFQ== 2026-02-13T02:41:39.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBjj45pkFYVKBAAMDj/LwBaJ0oIt2mWzGKaFQ=="}' 2026-02-13T02:41:39.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e5f0f6d4-6c65-44e3-97a9-6a92a2c2ad87 -i td/osd-scrub-test/1/new.json 2026-02-13T02:41:39.769 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:41:39.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/1/new.json 2026-02-13T02:41:39.779 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQBjj45pkFYVKBAAMDj/LwBaJ0oIt2mWzGKaFQ== --osd-uuid e5f0f6d4-6c65-44e3-97a9-6a92a2c2ad87 2026-02-13T02:41:39.795 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:39.790+0000 7f0da997aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:41:39.797 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:39.790+0000 7f0da997aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:41:39.798 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:39.790+0000 7f0da997aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:41:39.798 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:39.794+0000 7f0da997aa40 -1 bdev(0x55613cc8ac00 td/osd-scrub-test/1/block) open stat got: (1) Operation not permitted 2026-02-13T02:41:39.799 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:39.794+0000 7f0da997aa40 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2026-02-13T02:41:40.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2026-02-13T02:41:40.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:41:40.463 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-13T02:41:40.463 INFO:tasks.workunit.client.0.trial112.stdout:adding osd1 key to auth repository 2026-02-13T02:41:40.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:41:40.672 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T02:41:40.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-13T02:41:40.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2026-02-13T02:41:40.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:41:40.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:41:40.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:41:40.696 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:40.690+0000 7fd81f674a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:41:40.697 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:40.690+0000 7fd81f674a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:41:40.699 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:40.694+0000 7fd81f674a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:41:40.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-13T02:41:40.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:41:40.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:41:40.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:41:40.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:41:40.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:41:40.834 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:41:40.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:41:40.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:41:40.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:41:40.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:41:41.020 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:41.014+0000 7fd81f674a40 -1 Falling back to public interface 2026-02-13T02:41:41.567 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:41.562+0000 7fd81f674a40 -1 osd.1 0 log_to_monitors true 2026-02-13T02:41:41.998 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:41:41.998 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:41:41.999 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:41:41.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:41:42.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:41:42.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:41:42.157 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/889381634,v1:127.0.0.1:6811/889381634] [v2:127.0.0.1:6812/889381634,v1:127.0.0.1:6813/889381634] exists,up e5f0f6d4-6c65-44e3-97a9-6a92a2c2ad87 2026-02-13T02:41:42.160 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:41:42.160 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:41:42.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:41:42.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:311: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:41:42.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:317: _scrub_abort: run_osd td/osd-scrub-test 2 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2026-02-13T02:41:42.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-13T02:41:42.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:41:42.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-13T02:41:42.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:41:42.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/2 2026-02-13T02:41:42.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-13T02:41:42.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:41:42.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:41:42.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:41:42.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2026-02-13T02:41:42.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2026-02-13T02:41:42.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:41:42.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:41:42.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-13T02:41:42.162 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:41:42.162 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:41:42.162 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:41:42.162 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:41:42.162 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:41:42.162 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:41:42.162 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:41:42.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:41:42.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:41:42.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:41:42.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:41:42.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-13T02:41:42.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-13T02:41:42.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:41:42.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:41:42.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:41:42.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:41:42.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:41:42.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2026-02-13T02:41:42.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/2 2026-02-13T02:41:42.164 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:41:42.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=46a7e74a-e981-4b0c-a3ab-cd02859c458f 2026-02-13T02:41:42.165 INFO:tasks.workunit.client.0.trial112.stdout:add osd2 46a7e74a-e981-4b0c-a3ab-cd02859c458f 2026-02-13T02:41:42.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 46a7e74a-e981-4b0c-a3ab-cd02859c458f' 2026-02-13T02:41:42.165 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:41:42.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBmj45poJ4RChAA1Sx5GsBdIwYXuRIqKjLg4A== 2026-02-13T02:41:42.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBmj45poJ4RChAA1Sx5GsBdIwYXuRIqKjLg4A=="}' 2026-02-13T02:41:42.174 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 46a7e74a-e981-4b0c-a3ab-cd02859c458f -i td/osd-scrub-test/2/new.json 2026-02-13T02:41:42.335 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:41:42.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/2/new.json 2026-02-13T02:41:42.347 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQBmj45poJ4RChAA1Sx5GsBdIwYXuRIqKjLg4A== --osd-uuid 46a7e74a-e981-4b0c-a3ab-cd02859c458f 2026-02-13T02:41:42.372 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:42.366+0000 7f0ff0fbca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:41:42.373 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:42.366+0000 7f0ff0fbca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:41:42.375 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:42.370+0000 7f0ff0fbca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:41:42.375 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:42.370+0000 7f0ff0fbca40 -1 bdev(0x55c97bfdec00 td/osd-scrub-test/2/block) open stat got: (1) Operation not permitted 2026-02-13T02:41:42.375 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:42.370+0000 7f0ff0fbca40 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2026-02-13T02:41:43.050 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2026-02-13T02:41:43.050 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:41:43.051 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-13T02:41:43.051 INFO:tasks.workunit.client.0.trial112.stdout:adding osd2 key to auth repository 2026-02-13T02:41:43.052 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:41:43.260 INFO:tasks.workunit.client.0.trial112.stdout:start osd.2 2026-02-13T02:41:43.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-13T02:41:43.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2026-02-13T02:41:43.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:41:43.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:41:43.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:41:43.284 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:43.278+0000 7f59a1430a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:41:43.285 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:43.278+0000 7f59a1430a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:41:43.286 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:43.282+0000 7f59a1430a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:41:43.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-13T02:41:43.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:41:43.419 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T02:41:43.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:41:43.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:41:43.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:41:43.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:41:43.420 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:41:43.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:41:43.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:41:43.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:41:43.640 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:43.634+0000 7f59a1430a40 -1 Falling back to public interface 2026-02-13T02:41:44.285 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:41:44.278+0000 7f59a1430a40 -1 osd.2 0 log_to_monitors true 2026-02-13T02:41:44.580 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:41:44.580 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:41:44.580 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:41:44.581 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:41:44.581 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:41:44.581 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:41:44.747 INFO:tasks.workunit.client.0.trial112.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3347917738,v1:127.0.0.1:6819/3347917738] [v2:127.0.0.1:6820/3347917738,v1:127.0.0.1:6821/3347917738] exists,up 46a7e74a-e981-4b0c-a3ab-cd02859c458f 2026-02-13T02:41:44.748 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:41:44.748 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:41:44.748 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:41:44.748 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:325: _scrub_abort: create_pool test 1 1 2026-02-13T02:41:44.748 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2026-02-13T02:41:44.956 INFO:tasks.workunit.client.0.trial112.stderr:pool 'test' created 2026-02-13T02:41:44.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:41:45.970 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:326: _scrub_abort: wait_for_clean 2026-02-13T02:41:45.970 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:41:45.970 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:41:45.970 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:41:45.971 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:41:45.972 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:41:45.972 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:41:45.972 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:41:45.972 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:41:45.972 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:41:46.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:41:46.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:41:46.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:41:46.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:41:46.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:41:46.076 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:41:46.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:41:46.241 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:41:46.241 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:41:46.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:41:46.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:41:46.241 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:41:46.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836488 2026-02-13T02:41:46.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836488 2026-02-13T02:41:46.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488' 2026-02-13T02:41:46.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:41:46.314 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:41:46.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705669 2026-02-13T02:41:46.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705669 2026-02-13T02:41:46.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-38654705669' 2026-02-13T02:41:46.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:41:46.387 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:41:46.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574851 2026-02-13T02:41:46.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574851 2026-02-13T02:41:46.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-38654705669 2-55834574851' 2026-02-13T02:41:46.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:41:46.460 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836488 2026-02-13T02:41:46.460 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:41:46.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:41:46.461 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836488 2026-02-13T02:41:46.461 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:41:46.463 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836488 2026-02-13T02:41:46.463 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2026-02-13T02:41:46.463 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836488 2026-02-13T02:41:46.464 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:41:46.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836488 2026-02-13T02:41:46.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:41:47.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:41:47.623 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:41:47.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836488 2026-02-13T02:41:47.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:41:47.785 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705669 2026-02-13T02:41:47.785 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:41:47.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:41:47.787 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705669 2026-02-13T02:41:47.787 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:41:47.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705669 2026-02-13T02:41:47.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705669' 2026-02-13T02:41:47.789 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705669 2026-02-13T02:41:47.790 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:41:47.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705670 -lt 38654705669 2026-02-13T02:41:47.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:41:47.958 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574851 2026-02-13T02:41:47.958 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:41:47.959 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:41:47.960 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574851 2026-02-13T02:41:47.960 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:41:47.961 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574851 2026-02-13T02:41:47.961 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574851 2026-02-13T02:41:47.962 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574851' 2026-02-13T02:41:47.962 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:41:48.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574852 -lt 55834574851 2026-02-13T02:41:48.130 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:41:48.131 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:41:48.131 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:41:48.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:41:48.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:41:48.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:41:48.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:41:48.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:41:48.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:41:48.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:41:48.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:41:48.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:41:48.478 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:41:48.478 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:41:48.478 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:41:48.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:41:48.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:41:48.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:41:48.668 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:327: _scrub_abort: ceph osd dump 2026-02-13T02:41:48.669 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:327: _scrub_abort: grep '^pool.*['\'']test['\'']' 2026-02-13T02:41:48.669 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:327: _scrub_abort: awk '{ print $2 }' 2026-02-13T02:41:48.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:327: _scrub_abort: poolid=1 2026-02-13T02:41:48.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:329: _scrub_abort: dd if=/dev/urandom of=testdata.508946 bs=1032 count=1 2026-02-13T02:41:48.832 INFO:tasks.workunit.client.0.trial112.stderr:1+0 records in 2026-02-13T02:41:48.832 INFO:tasks.workunit.client.0.trial112.stderr:1+0 records out 2026-02-13T02:41:48.832 INFO:tasks.workunit.client.0.trial112.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 5.6355e-05 s, 18.3 MB/s 2026-02-13T02:41:48.833 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: seq 1 1000 2026-02-13T02:41:48.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:48.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj1 testdata.508946 2026-02-13T02:41:48.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:48.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj2 testdata.508946 2026-02-13T02:41:48.877 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:48.878 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj3 testdata.508946 2026-02-13T02:41:48.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:48.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj4 testdata.508946 2026-02-13T02:41:48.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:48.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj5 testdata.508946 2026-02-13T02:41:48.944 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:48.944 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj6 testdata.508946 2026-02-13T02:41:48.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:48.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj7 testdata.508946 2026-02-13T02:41:48.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:48.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj8 testdata.508946 2026-02-13T02:41:49.017 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.017 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj9 testdata.508946 2026-02-13T02:41:49.042 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.042 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj10 testdata.508946 2026-02-13T02:41:49.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj11 testdata.508946 2026-02-13T02:41:49.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj12 testdata.508946 2026-02-13T02:41:49.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj13 testdata.508946 2026-02-13T02:41:49.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj14 testdata.508946 2026-02-13T02:41:49.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj15 testdata.508946 2026-02-13T02:41:49.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.190 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj16 testdata.508946 2026-02-13T02:41:49.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj17 testdata.508946 2026-02-13T02:41:49.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj18 testdata.508946 2026-02-13T02:41:49.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj19 testdata.508946 2026-02-13T02:41:49.280 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.280 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj20 testdata.508946 2026-02-13T02:41:49.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj21 testdata.508946 2026-02-13T02:41:49.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj22 testdata.508946 2026-02-13T02:41:49.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj23 testdata.508946 2026-02-13T02:41:49.376 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.376 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj24 testdata.508946 2026-02-13T02:41:49.401 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.401 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj25 testdata.508946 2026-02-13T02:41:49.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj26 testdata.508946 2026-02-13T02:41:49.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj27 testdata.508946 2026-02-13T02:41:49.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.468 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj28 testdata.508946 2026-02-13T02:41:49.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj29 testdata.508946 2026-02-13T02:41:49.516 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.516 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj30 testdata.508946 2026-02-13T02:41:49.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj31 testdata.508946 2026-02-13T02:41:49.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj32 testdata.508946 2026-02-13T02:41:49.581 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.581 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj33 testdata.508946 2026-02-13T02:41:49.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj34 testdata.508946 2026-02-13T02:41:49.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj35 testdata.508946 2026-02-13T02:41:49.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj36 testdata.508946 2026-02-13T02:41:49.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj37 testdata.508946 2026-02-13T02:41:49.692 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.692 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj38 testdata.508946 2026-02-13T02:41:49.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj39 testdata.508946 2026-02-13T02:41:49.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj40 testdata.508946 2026-02-13T02:41:49.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj41 testdata.508946 2026-02-13T02:41:49.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj42 testdata.508946 2026-02-13T02:41:49.806 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.806 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj43 testdata.508946 2026-02-13T02:41:49.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj44 testdata.508946 2026-02-13T02:41:49.854 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.854 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj45 testdata.508946 2026-02-13T02:41:49.878 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.878 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj46 testdata.508946 2026-02-13T02:41:49.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj47 testdata.508946 2026-02-13T02:41:49.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj48 testdata.508946 2026-02-13T02:41:49.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj49 testdata.508946 2026-02-13T02:41:49.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj50 testdata.508946 2026-02-13T02:41:49.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:49.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj51 testdata.508946 2026-02-13T02:41:50.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj52 testdata.508946 2026-02-13T02:41:50.042 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.042 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj53 testdata.508946 2026-02-13T02:41:50.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj54 testdata.508946 2026-02-13T02:41:50.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj55 testdata.508946 2026-02-13T02:41:50.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj56 testdata.508946 2026-02-13T02:41:50.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj57 testdata.508946 2026-02-13T02:41:50.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj58 testdata.508946 2026-02-13T02:41:50.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj59 testdata.508946 2026-02-13T02:41:50.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj60 testdata.508946 2026-02-13T02:41:50.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj61 testdata.508946 2026-02-13T02:41:50.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj62 testdata.508946 2026-02-13T02:41:50.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj63 testdata.508946 2026-02-13T02:41:50.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj64 testdata.508946 2026-02-13T02:41:50.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj65 testdata.508946 2026-02-13T02:41:50.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj66 testdata.508946 2026-02-13T02:41:50.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj67 testdata.508946 2026-02-13T02:41:50.381 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.381 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj68 testdata.508946 2026-02-13T02:41:50.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj69 testdata.508946 2026-02-13T02:41:50.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj70 testdata.508946 2026-02-13T02:41:50.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj71 testdata.508946 2026-02-13T02:41:50.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj72 testdata.508946 2026-02-13T02:41:50.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj73 testdata.508946 2026-02-13T02:41:50.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj74 testdata.508946 2026-02-13T02:41:50.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj75 testdata.508946 2026-02-13T02:41:50.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj76 testdata.508946 2026-02-13T02:41:50.594 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj77 testdata.508946 2026-02-13T02:41:50.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj78 testdata.508946 2026-02-13T02:41:50.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj79 testdata.508946 2026-02-13T02:41:50.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj80 testdata.508946 2026-02-13T02:41:50.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj81 testdata.508946 2026-02-13T02:41:50.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj82 testdata.508946 2026-02-13T02:41:50.737 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj83 testdata.508946 2026-02-13T02:41:50.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.762 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj84 testdata.508946 2026-02-13T02:41:50.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj85 testdata.508946 2026-02-13T02:41:50.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj86 testdata.508946 2026-02-13T02:41:50.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj87 testdata.508946 2026-02-13T02:41:50.854 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.854 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj88 testdata.508946 2026-02-13T02:41:50.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj89 testdata.508946 2026-02-13T02:41:50.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj90 testdata.508946 2026-02-13T02:41:50.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj91 testdata.508946 2026-02-13T02:41:50.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj92 testdata.508946 2026-02-13T02:41:50.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj93 testdata.508946 2026-02-13T02:41:50.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:50.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj94 testdata.508946 2026-02-13T02:41:51.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj95 testdata.508946 2026-02-13T02:41:51.037 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.037 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj96 testdata.508946 2026-02-13T02:41:51.054 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.055 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj97 testdata.508946 2026-02-13T02:41:51.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj98 testdata.508946 2026-02-13T02:41:51.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj99 testdata.508946 2026-02-13T02:41:51.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj100 testdata.508946 2026-02-13T02:41:51.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj101 testdata.508946 2026-02-13T02:41:51.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.173 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj102 testdata.508946 2026-02-13T02:41:51.196 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj103 testdata.508946 2026-02-13T02:41:51.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj104 testdata.508946 2026-02-13T02:41:51.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.242 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj105 testdata.508946 2026-02-13T02:41:51.266 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.266 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj106 testdata.508946 2026-02-13T02:41:51.290 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.290 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj107 testdata.508946 2026-02-13T02:41:51.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj108 testdata.508946 2026-02-13T02:41:51.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj109 testdata.508946 2026-02-13T02:41:51.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj110 testdata.508946 2026-02-13T02:41:51.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj111 testdata.508946 2026-02-13T02:41:51.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj112 testdata.508946 2026-02-13T02:41:51.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj113 testdata.508946 2026-02-13T02:41:51.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.463 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj114 testdata.508946 2026-02-13T02:41:51.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj115 testdata.508946 2026-02-13T02:41:51.511 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.511 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj116 testdata.508946 2026-02-13T02:41:51.536 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.536 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj117 testdata.508946 2026-02-13T02:41:51.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj118 testdata.508946 2026-02-13T02:41:51.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj119 testdata.508946 2026-02-13T02:41:51.611 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.611 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj120 testdata.508946 2026-02-13T02:41:51.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj121 testdata.508946 2026-02-13T02:41:51.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj122 testdata.508946 2026-02-13T02:41:51.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj123 testdata.508946 2026-02-13T02:41:51.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj124 testdata.508946 2026-02-13T02:41:51.735 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.735 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj125 testdata.508946 2026-02-13T02:41:51.760 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.760 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj126 testdata.508946 2026-02-13T02:41:51.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj127 testdata.508946 2026-02-13T02:41:51.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj128 testdata.508946 2026-02-13T02:41:51.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.835 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj129 testdata.508946 2026-02-13T02:41:51.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj130 testdata.508946 2026-02-13T02:41:51.876 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.877 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj131 testdata.508946 2026-02-13T02:41:51.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj132 testdata.508946 2026-02-13T02:41:51.925 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.925 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj133 testdata.508946 2026-02-13T02:41:51.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.950 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj134 testdata.508946 2026-02-13T02:41:51.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:51.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj135 testdata.508946 2026-02-13T02:41:51.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj136 testdata.508946 2026-02-13T02:41:52.024 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.025 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj137 testdata.508946 2026-02-13T02:41:52.042 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj138 testdata.508946 2026-02-13T02:41:52.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj139 testdata.508946 2026-02-13T02:41:52.079 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.079 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj140 testdata.508946 2026-02-13T02:41:52.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj141 testdata.508946 2026-02-13T02:41:52.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj142 testdata.508946 2026-02-13T02:41:52.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj143 testdata.508946 2026-02-13T02:41:52.175 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.175 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj144 testdata.508946 2026-02-13T02:41:52.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj145 testdata.508946 2026-02-13T02:41:52.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj146 testdata.508946 2026-02-13T02:41:52.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj147 testdata.508946 2026-02-13T02:41:52.269 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.269 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj148 testdata.508946 2026-02-13T02:41:52.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj149 testdata.508946 2026-02-13T02:41:52.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj150 testdata.508946 2026-02-13T02:41:52.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj151 testdata.508946 2026-02-13T02:41:52.358 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.358 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj152 testdata.508946 2026-02-13T02:41:52.382 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.382 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj153 testdata.508946 2026-02-13T02:41:52.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj154 testdata.508946 2026-02-13T02:41:52.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj155 testdata.508946 2026-02-13T02:41:52.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj156 testdata.508946 2026-02-13T02:41:52.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj157 testdata.508946 2026-02-13T02:41:52.497 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.497 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj158 testdata.508946 2026-02-13T02:41:52.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj159 testdata.508946 2026-02-13T02:41:52.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj160 testdata.508946 2026-02-13T02:41:52.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj161 testdata.508946 2026-02-13T02:41:52.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj162 testdata.508946 2026-02-13T02:41:52.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.614 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj163 testdata.508946 2026-02-13T02:41:52.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj164 testdata.508946 2026-02-13T02:41:52.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj165 testdata.508946 2026-02-13T02:41:52.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj166 testdata.508946 2026-02-13T02:41:52.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj167 testdata.508946 2026-02-13T02:41:52.735 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.735 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj168 testdata.508946 2026-02-13T02:41:52.760 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.760 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj169 testdata.508946 2026-02-13T02:41:52.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj170 testdata.508946 2026-02-13T02:41:52.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj171 testdata.508946 2026-02-13T02:41:52.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj172 testdata.508946 2026-02-13T02:41:52.845 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.845 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj173 testdata.508946 2026-02-13T02:41:52.869 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.869 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj174 testdata.508946 2026-02-13T02:41:52.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj175 testdata.508946 2026-02-13T02:41:52.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj176 testdata.508946 2026-02-13T02:41:52.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj177 testdata.508946 2026-02-13T02:41:52.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj178 testdata.508946 2026-02-13T02:41:52.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:52.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj179 testdata.508946 2026-02-13T02:41:53.016 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.016 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj180 testdata.508946 2026-02-13T02:41:53.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj181 testdata.508946 2026-02-13T02:41:53.065 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.065 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj182 testdata.508946 2026-02-13T02:41:53.089 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.089 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj183 testdata.508946 2026-02-13T02:41:53.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj184 testdata.508946 2026-02-13T02:41:53.138 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.138 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj185 testdata.508946 2026-02-13T02:41:53.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj186 testdata.508946 2026-02-13T02:41:53.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj187 testdata.508946 2026-02-13T02:41:53.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj188 testdata.508946 2026-02-13T02:41:53.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj189 testdata.508946 2026-02-13T02:41:53.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj190 testdata.508946 2026-02-13T02:41:53.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj191 testdata.508946 2026-02-13T02:41:53.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj192 testdata.508946 2026-02-13T02:41:53.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj193 testdata.508946 2026-02-13T02:41:53.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj194 testdata.508946 2026-02-13T02:41:53.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj195 testdata.508946 2026-02-13T02:41:53.403 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.403 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj196 testdata.508946 2026-02-13T02:41:53.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj197 testdata.508946 2026-02-13T02:41:53.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj198 testdata.508946 2026-02-13T02:41:53.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj199 testdata.508946 2026-02-13T02:41:53.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj200 testdata.508946 2026-02-13T02:41:53.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj201 testdata.508946 2026-02-13T02:41:53.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj202 testdata.508946 2026-02-13T02:41:53.574 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.575 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj203 testdata.508946 2026-02-13T02:41:53.599 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.600 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj204 testdata.508946 2026-02-13T02:41:53.624 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.624 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj205 testdata.508946 2026-02-13T02:41:53.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj206 testdata.508946 2026-02-13T02:41:53.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj207 testdata.508946 2026-02-13T02:41:53.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj208 testdata.508946 2026-02-13T02:41:53.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj209 testdata.508946 2026-02-13T02:41:53.739 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.739 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj210 testdata.508946 2026-02-13T02:41:53.764 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.764 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj211 testdata.508946 2026-02-13T02:41:53.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj212 testdata.508946 2026-02-13T02:41:53.806 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.806 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj213 testdata.508946 2026-02-13T02:41:53.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj214 testdata.508946 2026-02-13T02:41:53.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj215 testdata.508946 2026-02-13T02:41:53.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj216 testdata.508946 2026-02-13T02:41:53.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj217 testdata.508946 2026-02-13T02:41:53.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj218 testdata.508946 2026-02-13T02:41:53.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj219 testdata.508946 2026-02-13T02:41:53.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj220 testdata.508946 2026-02-13T02:41:53.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:53.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj221 testdata.508946 2026-02-13T02:41:54.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj222 testdata.508946 2026-02-13T02:41:54.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj223 testdata.508946 2026-02-13T02:41:54.052 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.053 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj224 testdata.508946 2026-02-13T02:41:54.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj225 testdata.508946 2026-02-13T02:41:54.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj226 testdata.508946 2026-02-13T02:41:54.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj227 testdata.508946 2026-02-13T02:41:54.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj228 testdata.508946 2026-02-13T02:41:54.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj229 testdata.508946 2026-02-13T02:41:54.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj230 testdata.508946 2026-02-13T02:41:54.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj231 testdata.508946 2026-02-13T02:41:54.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj232 testdata.508946 2026-02-13T02:41:54.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj233 testdata.508946 2026-02-13T02:41:54.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj234 testdata.508946 2026-02-13T02:41:54.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj235 testdata.508946 2026-02-13T02:41:54.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj236 testdata.508946 2026-02-13T02:41:54.339 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj237 testdata.508946 2026-02-13T02:41:54.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj238 testdata.508946 2026-02-13T02:41:54.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj239 testdata.508946 2026-02-13T02:41:54.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.413 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj240 testdata.508946 2026-02-13T02:41:54.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj241 testdata.508946 2026-02-13T02:41:54.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj242 testdata.508946 2026-02-13T02:41:54.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj243 testdata.508946 2026-02-13T02:41:54.510 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.510 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj244 testdata.508946 2026-02-13T02:41:54.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj245 testdata.508946 2026-02-13T02:41:54.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj246 testdata.508946 2026-02-13T02:41:54.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.578 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj247 testdata.508946 2026-02-13T02:41:54.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj248 testdata.508946 2026-02-13T02:41:54.627 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.627 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj249 testdata.508946 2026-02-13T02:41:54.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj250 testdata.508946 2026-02-13T02:41:54.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.677 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj251 testdata.508946 2026-02-13T02:41:54.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj252 testdata.508946 2026-02-13T02:41:54.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj253 testdata.508946 2026-02-13T02:41:54.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj254 testdata.508946 2026-02-13T02:41:54.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj255 testdata.508946 2026-02-13T02:41:54.801 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.801 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj256 testdata.508946 2026-02-13T02:41:54.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj257 testdata.508946 2026-02-13T02:41:54.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj258 testdata.508946 2026-02-13T02:41:54.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.876 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj259 testdata.508946 2026-02-13T02:41:54.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj260 testdata.508946 2026-02-13T02:41:54.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj261 testdata.508946 2026-02-13T02:41:54.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj262 testdata.508946 2026-02-13T02:41:54.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj263 testdata.508946 2026-02-13T02:41:54.988 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:54.988 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj264 testdata.508946 2026-02-13T02:41:55.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj265 testdata.508946 2026-02-13T02:41:55.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj266 testdata.508946 2026-02-13T02:41:55.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj267 testdata.508946 2026-02-13T02:41:55.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj268 testdata.508946 2026-02-13T02:41:55.105 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj269 testdata.508946 2026-02-13T02:41:55.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj270 testdata.508946 2026-02-13T02:41:55.154 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.154 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj271 testdata.508946 2026-02-13T02:41:55.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj272 testdata.508946 2026-02-13T02:41:55.203 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.203 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj273 testdata.508946 2026-02-13T02:41:55.228 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.228 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj274 testdata.508946 2026-02-13T02:41:55.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj275 testdata.508946 2026-02-13T02:41:55.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.276 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj276 testdata.508946 2026-02-13T02:41:55.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj277 testdata.508946 2026-02-13T02:41:55.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj278 testdata.508946 2026-02-13T02:41:55.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.351 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj279 testdata.508946 2026-02-13T02:41:55.369 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.369 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj280 testdata.508946 2026-02-13T02:41:55.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.393 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj281 testdata.508946 2026-02-13T02:41:55.416 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj282 testdata.508946 2026-02-13T02:41:55.442 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.442 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj283 testdata.508946 2026-02-13T02:41:55.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj284 testdata.508946 2026-02-13T02:41:55.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj285 testdata.508946 2026-02-13T02:41:55.508 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.508 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj286 testdata.508946 2026-02-13T02:41:55.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj287 testdata.508946 2026-02-13T02:41:55.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj288 testdata.508946 2026-02-13T02:41:55.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.579 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj289 testdata.508946 2026-02-13T02:41:55.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.604 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj290 testdata.508946 2026-02-13T02:41:55.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj291 testdata.508946 2026-02-13T02:41:55.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj292 testdata.508946 2026-02-13T02:41:55.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj293 testdata.508946 2026-02-13T02:41:55.694 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.694 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj294 testdata.508946 2026-02-13T02:41:55.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj295 testdata.508946 2026-02-13T02:41:55.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj296 testdata.508946 2026-02-13T02:41:55.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj297 testdata.508946 2026-02-13T02:41:55.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj298 testdata.508946 2026-02-13T02:41:55.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj299 testdata.508946 2026-02-13T02:41:55.836 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.836 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj300 testdata.508946 2026-02-13T02:41:55.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj301 testdata.508946 2026-02-13T02:41:55.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj302 testdata.508946 2026-02-13T02:41:55.904 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.904 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj303 testdata.508946 2026-02-13T02:41:55.929 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.929 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj304 testdata.508946 2026-02-13T02:41:55.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj305 testdata.508946 2026-02-13T02:41:55.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:55.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj306 testdata.508946 2026-02-13T02:41:56.001 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.001 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj307 testdata.508946 2026-02-13T02:41:56.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj308 testdata.508946 2026-02-13T02:41:56.050 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.050 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj309 testdata.508946 2026-02-13T02:41:56.074 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.074 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj310 testdata.508946 2026-02-13T02:41:56.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj311 testdata.508946 2026-02-13T02:41:56.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj312 testdata.508946 2026-02-13T02:41:56.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj313 testdata.508946 2026-02-13T02:41:56.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj314 testdata.508946 2026-02-13T02:41:56.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj315 testdata.508946 2026-02-13T02:41:56.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj316 testdata.508946 2026-02-13T02:41:56.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj317 testdata.508946 2026-02-13T02:41:56.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj318 testdata.508946 2026-02-13T02:41:56.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj319 testdata.508946 2026-02-13T02:41:56.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj320 testdata.508946 2026-02-13T02:41:56.336 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj321 testdata.508946 2026-02-13T02:41:56.361 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj322 testdata.508946 2026-02-13T02:41:56.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj323 testdata.508946 2026-02-13T02:41:56.410 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj324 testdata.508946 2026-02-13T02:41:56.435 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.435 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj325 testdata.508946 2026-02-13T02:41:56.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.459 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj326 testdata.508946 2026-02-13T02:41:56.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj327 testdata.508946 2026-02-13T02:41:56.508 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.508 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj328 testdata.508946 2026-02-13T02:41:56.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj329 testdata.508946 2026-02-13T02:41:56.557 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.557 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj330 testdata.508946 2026-02-13T02:41:56.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj331 testdata.508946 2026-02-13T02:41:56.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj332 testdata.508946 2026-02-13T02:41:56.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.631 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj333 testdata.508946 2026-02-13T02:41:56.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj334 testdata.508946 2026-02-13T02:41:56.680 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.680 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj335 testdata.508946 2026-02-13T02:41:56.704 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.704 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj336 testdata.508946 2026-02-13T02:41:56.729 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.729 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj337 testdata.508946 2026-02-13T02:41:56.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj338 testdata.508946 2026-02-13T02:41:56.771 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.771 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj339 testdata.508946 2026-02-13T02:41:56.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj340 testdata.508946 2026-02-13T02:41:56.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj341 testdata.508946 2026-02-13T02:41:56.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj342 testdata.508946 2026-02-13T02:41:56.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj343 testdata.508946 2026-02-13T02:41:56.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj344 testdata.508946 2026-02-13T02:41:56.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj345 testdata.508946 2026-02-13T02:41:56.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj346 testdata.508946 2026-02-13T02:41:56.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj347 testdata.508946 2026-02-13T02:41:56.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:56.985 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj348 testdata.508946 2026-02-13T02:41:57.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj349 testdata.508946 2026-02-13T02:41:57.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj350 testdata.508946 2026-02-13T02:41:57.058 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.058 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj351 testdata.508946 2026-02-13T02:41:57.082 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.082 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj352 testdata.508946 2026-02-13T02:41:57.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.107 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj353 testdata.508946 2026-02-13T02:41:57.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj354 testdata.508946 2026-02-13T02:41:57.155 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.155 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj355 testdata.508946 2026-02-13T02:41:57.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.180 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj356 testdata.508946 2026-02-13T02:41:57.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj357 testdata.508946 2026-02-13T02:41:57.228 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.229 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj358 testdata.508946 2026-02-13T02:41:57.253 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.253 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj359 testdata.508946 2026-02-13T02:41:57.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.278 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj360 testdata.508946 2026-02-13T02:41:57.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj361 testdata.508946 2026-02-13T02:41:57.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj362 testdata.508946 2026-02-13T02:41:57.352 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj363 testdata.508946 2026-02-13T02:41:57.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj364 testdata.508946 2026-02-13T02:41:57.403 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.403 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj365 testdata.508946 2026-02-13T02:41:57.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj366 testdata.508946 2026-02-13T02:41:57.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj367 testdata.508946 2026-02-13T02:41:57.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj368 testdata.508946 2026-02-13T02:41:57.502 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.502 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj369 testdata.508946 2026-02-13T02:41:57.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj370 testdata.508946 2026-02-13T02:41:57.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj371 testdata.508946 2026-02-13T02:41:57.574 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.574 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj372 testdata.508946 2026-02-13T02:41:57.598 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.599 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj373 testdata.508946 2026-02-13T02:41:57.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj374 testdata.508946 2026-02-13T02:41:57.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj375 testdata.508946 2026-02-13T02:41:57.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj376 testdata.508946 2026-02-13T02:41:57.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj377 testdata.508946 2026-02-13T02:41:57.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj378 testdata.508946 2026-02-13T02:41:57.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj379 testdata.508946 2026-02-13T02:41:57.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj380 testdata.508946 2026-02-13T02:41:57.775 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.775 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj381 testdata.508946 2026-02-13T02:41:57.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj382 testdata.508946 2026-02-13T02:41:57.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.819 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj383 testdata.508946 2026-02-13T02:41:57.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.837 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj384 testdata.508946 2026-02-13T02:41:57.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj385 testdata.508946 2026-02-13T02:41:57.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj386 testdata.508946 2026-02-13T02:41:57.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.909 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj387 testdata.508946 2026-02-13T02:41:57.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj388 testdata.508946 2026-02-13T02:41:57.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj389 testdata.508946 2026-02-13T02:41:57.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:57.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj390 testdata.508946 2026-02-13T02:41:58.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj391 testdata.508946 2026-02-13T02:41:58.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj392 testdata.508946 2026-02-13T02:41:58.057 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.057 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj393 testdata.508946 2026-02-13T02:41:58.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.082 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj394 testdata.508946 2026-02-13T02:41:58.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj395 testdata.508946 2026-02-13T02:41:58.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.130 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj396 testdata.508946 2026-02-13T02:41:58.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj397 testdata.508946 2026-02-13T02:41:58.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj398 testdata.508946 2026-02-13T02:41:58.195 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.195 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj399 testdata.508946 2026-02-13T02:41:58.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj400 testdata.508946 2026-02-13T02:41:58.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj401 testdata.508946 2026-02-13T02:41:58.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj402 testdata.508946 2026-02-13T02:41:58.280 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.280 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj403 testdata.508946 2026-02-13T02:41:58.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj404 testdata.508946 2026-02-13T02:41:58.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj405 testdata.508946 2026-02-13T02:41:58.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj406 testdata.508946 2026-02-13T02:41:58.369 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.369 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj407 testdata.508946 2026-02-13T02:41:58.393 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.393 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj408 testdata.508946 2026-02-13T02:41:58.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj409 testdata.508946 2026-02-13T02:41:58.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.435 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj410 testdata.508946 2026-02-13T02:41:58.456 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj411 testdata.508946 2026-02-13T02:41:58.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj412 testdata.508946 2026-02-13T02:41:58.507 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.507 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj413 testdata.508946 2026-02-13T02:41:58.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj414 testdata.508946 2026-02-13T02:41:58.556 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.556 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj415 testdata.508946 2026-02-13T02:41:58.580 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.580 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj416 testdata.508946 2026-02-13T02:41:58.601 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.601 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj417 testdata.508946 2026-02-13T02:41:58.624 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.624 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj418 testdata.508946 2026-02-13T02:41:58.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj419 testdata.508946 2026-02-13T02:41:58.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj420 testdata.508946 2026-02-13T02:41:58.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.695 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj421 testdata.508946 2026-02-13T02:41:58.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj422 testdata.508946 2026-02-13T02:41:58.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj423 testdata.508946 2026-02-13T02:41:58.768 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.769 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj424 testdata.508946 2026-02-13T02:41:58.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj425 testdata.508946 2026-02-13T02:41:58.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj426 testdata.508946 2026-02-13T02:41:58.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj427 testdata.508946 2026-02-13T02:41:58.866 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.866 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj428 testdata.508946 2026-02-13T02:41:58.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj429 testdata.508946 2026-02-13T02:41:58.915 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.915 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj430 testdata.508946 2026-02-13T02:41:58.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj431 testdata.508946 2026-02-13T02:41:58.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj432 testdata.508946 2026-02-13T02:41:58.988 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:58.988 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj433 testdata.508946 2026-02-13T02:41:59.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj434 testdata.508946 2026-02-13T02:41:59.037 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.037 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj435 testdata.508946 2026-02-13T02:41:59.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj436 testdata.508946 2026-02-13T02:41:59.086 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj437 testdata.508946 2026-02-13T02:41:59.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj438 testdata.508946 2026-02-13T02:41:59.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj439 testdata.508946 2026-02-13T02:41:59.160 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.160 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj440 testdata.508946 2026-02-13T02:41:59.184 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.184 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj441 testdata.508946 2026-02-13T02:41:59.202 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.202 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj442 testdata.508946 2026-02-13T02:41:59.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj443 testdata.508946 2026-02-13T02:41:59.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj444 testdata.508946 2026-02-13T02:41:59.267 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj445 testdata.508946 2026-02-13T02:41:59.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj446 testdata.508946 2026-02-13T02:41:59.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj447 testdata.508946 2026-02-13T02:41:59.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj448 testdata.508946 2026-02-13T02:41:59.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.350 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj449 testdata.508946 2026-02-13T02:41:59.372 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.372 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj450 testdata.508946 2026-02-13T02:41:59.395 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.395 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj451 testdata.508946 2026-02-13T02:41:59.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj452 testdata.508946 2026-02-13T02:41:59.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj453 testdata.508946 2026-02-13T02:41:59.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj454 testdata.508946 2026-02-13T02:41:59.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj455 testdata.508946 2026-02-13T02:41:59.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj456 testdata.508946 2026-02-13T02:41:59.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj457 testdata.508946 2026-02-13T02:41:59.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj458 testdata.508946 2026-02-13T02:41:59.575 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.575 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj459 testdata.508946 2026-02-13T02:41:59.600 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.600 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj460 testdata.508946 2026-02-13T02:41:59.625 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.625 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj461 testdata.508946 2026-02-13T02:41:59.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj462 testdata.508946 2026-02-13T02:41:59.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj463 testdata.508946 2026-02-13T02:41:59.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj464 testdata.508946 2026-02-13T02:41:59.723 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.723 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj465 testdata.508946 2026-02-13T02:41:59.747 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.748 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj466 testdata.508946 2026-02-13T02:41:59.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj467 testdata.508946 2026-02-13T02:41:59.797 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.797 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj468 testdata.508946 2026-02-13T02:41:59.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj469 testdata.508946 2026-02-13T02:41:59.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.847 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj470 testdata.508946 2026-02-13T02:41:59.871 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.871 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj471 testdata.508946 2026-02-13T02:41:59.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj472 testdata.508946 2026-02-13T02:41:59.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj473 testdata.508946 2026-02-13T02:41:59.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj474 testdata.508946 2026-02-13T02:41:59.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj475 testdata.508946 2026-02-13T02:41:59.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:41:59.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj476 testdata.508946 2026-02-13T02:42:00.017 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.017 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj477 testdata.508946 2026-02-13T02:42:00.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj478 testdata.508946 2026-02-13T02:42:00.052 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.052 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj479 testdata.508946 2026-02-13T02:42:00.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj480 testdata.508946 2026-02-13T02:42:00.099 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.099 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj481 testdata.508946 2026-02-13T02:42:00.123 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.123 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj482 testdata.508946 2026-02-13T02:42:00.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj483 testdata.508946 2026-02-13T02:42:00.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj484 testdata.508946 2026-02-13T02:42:00.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj485 testdata.508946 2026-02-13T02:42:00.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj486 testdata.508946 2026-02-13T02:42:00.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj487 testdata.508946 2026-02-13T02:42:00.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj488 testdata.508946 2026-02-13T02:42:00.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj489 testdata.508946 2026-02-13T02:42:00.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj490 testdata.508946 2026-02-13T02:42:00.331 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.332 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj491 testdata.508946 2026-02-13T02:42:00.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj492 testdata.508946 2026-02-13T02:42:00.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj493 testdata.508946 2026-02-13T02:42:00.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj494 testdata.508946 2026-02-13T02:42:00.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj495 testdata.508946 2026-02-13T02:42:00.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.451 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj496 testdata.508946 2026-02-13T02:42:00.474 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.474 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj497 testdata.508946 2026-02-13T02:42:00.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj498 testdata.508946 2026-02-13T02:42:00.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj499 testdata.508946 2026-02-13T02:42:00.536 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.537 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj500 testdata.508946 2026-02-13T02:42:00.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj501 testdata.508946 2026-02-13T02:42:00.585 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.585 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj502 testdata.508946 2026-02-13T02:42:00.610 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.610 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj503 testdata.508946 2026-02-13T02:42:00.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj504 testdata.508946 2026-02-13T02:42:00.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj505 testdata.508946 2026-02-13T02:42:00.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj506 testdata.508946 2026-02-13T02:42:00.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj507 testdata.508946 2026-02-13T02:42:00.720 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.720 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj508 testdata.508946 2026-02-13T02:42:00.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj509 testdata.508946 2026-02-13T02:42:00.767 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.768 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj510 testdata.508946 2026-02-13T02:42:00.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj511 testdata.508946 2026-02-13T02:42:00.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj512 testdata.508946 2026-02-13T02:42:00.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj513 testdata.508946 2026-02-13T02:42:00.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj514 testdata.508946 2026-02-13T02:42:00.886 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.886 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj515 testdata.508946 2026-02-13T02:42:00.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj516 testdata.508946 2026-02-13T02:42:00.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj517 testdata.508946 2026-02-13T02:42:00.959 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.959 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj518 testdata.508946 2026-02-13T02:42:00.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:00.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj519 testdata.508946 2026-02-13T02:42:01.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj520 testdata.508946 2026-02-13T02:42:01.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj521 testdata.508946 2026-02-13T02:42:01.050 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.050 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj522 testdata.508946 2026-02-13T02:42:01.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj523 testdata.508946 2026-02-13T02:42:01.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj524 testdata.508946 2026-02-13T02:42:01.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj525 testdata.508946 2026-02-13T02:42:01.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.134 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj526 testdata.508946 2026-02-13T02:42:01.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj527 testdata.508946 2026-02-13T02:42:01.182 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.182 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj528 testdata.508946 2026-02-13T02:42:01.206 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.206 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj529 testdata.508946 2026-02-13T02:42:01.230 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.230 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj530 testdata.508946 2026-02-13T02:42:01.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj531 testdata.508946 2026-02-13T02:42:01.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj532 testdata.508946 2026-02-13T02:42:01.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj533 testdata.508946 2026-02-13T02:42:01.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj534 testdata.508946 2026-02-13T02:42:01.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj535 testdata.508946 2026-02-13T02:42:01.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj536 testdata.508946 2026-02-13T02:42:01.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj537 testdata.508946 2026-02-13T02:42:01.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj538 testdata.508946 2026-02-13T02:42:01.435 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.435 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj539 testdata.508946 2026-02-13T02:42:01.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj540 testdata.508946 2026-02-13T02:42:01.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj541 testdata.508946 2026-02-13T02:42:01.508 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.508 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj542 testdata.508946 2026-02-13T02:42:01.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj543 testdata.508946 2026-02-13T02:42:01.557 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.557 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj544 testdata.508946 2026-02-13T02:42:01.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.582 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj545 testdata.508946 2026-02-13T02:42:01.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj546 testdata.508946 2026-02-13T02:42:01.625 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.625 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj547 testdata.508946 2026-02-13T02:42:01.648 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj548 testdata.508946 2026-02-13T02:42:01.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.674 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj549 testdata.508946 2026-02-13T02:42:01.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj550 testdata.508946 2026-02-13T02:42:01.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj551 testdata.508946 2026-02-13T02:42:01.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj552 testdata.508946 2026-02-13T02:42:01.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj553 testdata.508946 2026-02-13T02:42:01.774 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.774 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj554 testdata.508946 2026-02-13T02:42:01.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj555 testdata.508946 2026-02-13T02:42:01.823 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.823 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj556 testdata.508946 2026-02-13T02:42:01.848 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.848 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj557 testdata.508946 2026-02-13T02:42:01.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj558 testdata.508946 2026-02-13T02:42:01.898 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.898 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj559 testdata.508946 2026-02-13T02:42:01.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj560 testdata.508946 2026-02-13T02:42:01.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj561 testdata.508946 2026-02-13T02:42:01.971 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.971 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj562 testdata.508946 2026-02-13T02:42:01.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:01.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj563 testdata.508946 2026-02-13T02:42:02.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.021 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj564 testdata.508946 2026-02-13T02:42:02.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj565 testdata.508946 2026-02-13T02:42:02.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj566 testdata.508946 2026-02-13T02:42:02.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj567 testdata.508946 2026-02-13T02:42:02.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj568 testdata.508946 2026-02-13T02:42:02.138 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.139 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj569 testdata.508946 2026-02-13T02:42:02.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj570 testdata.508946 2026-02-13T02:42:02.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.179 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj571 testdata.508946 2026-02-13T02:42:02.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj572 testdata.508946 2026-02-13T02:42:02.223 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.223 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj573 testdata.508946 2026-02-13T02:42:02.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj574 testdata.508946 2026-02-13T02:42:02.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj575 testdata.508946 2026-02-13T02:42:02.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj576 testdata.508946 2026-02-13T02:42:02.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.311 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj577 testdata.508946 2026-02-13T02:42:02.336 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.336 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj578 testdata.508946 2026-02-13T02:42:02.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj579 testdata.508946 2026-02-13T02:42:02.385 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.385 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj580 testdata.508946 2026-02-13T02:42:02.410 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.410 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj581 testdata.508946 2026-02-13T02:42:02.435 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.435 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj582 testdata.508946 2026-02-13T02:42:02.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj583 testdata.508946 2026-02-13T02:42:02.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj584 testdata.508946 2026-02-13T02:42:02.509 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.509 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj585 testdata.508946 2026-02-13T02:42:02.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj586 testdata.508946 2026-02-13T02:42:02.558 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj587 testdata.508946 2026-02-13T02:42:02.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.584 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj588 testdata.508946 2026-02-13T02:42:02.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.608 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj589 testdata.508946 2026-02-13T02:42:02.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj590 testdata.508946 2026-02-13T02:42:02.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj591 testdata.508946 2026-02-13T02:42:02.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj592 testdata.508946 2026-02-13T02:42:02.707 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.707 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj593 testdata.508946 2026-02-13T02:42:02.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj594 testdata.508946 2026-02-13T02:42:02.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj595 testdata.508946 2026-02-13T02:42:02.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.781 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj596 testdata.508946 2026-02-13T02:42:02.806 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.806 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj597 testdata.508946 2026-02-13T02:42:02.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj598 testdata.508946 2026-02-13T02:42:02.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj599 testdata.508946 2026-02-13T02:42:02.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj600 testdata.508946 2026-02-13T02:42:02.904 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.904 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj601 testdata.508946 2026-02-13T02:42:02.929 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.929 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj602 testdata.508946 2026-02-13T02:42:02.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj603 testdata.508946 2026-02-13T02:42:02.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:02.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj604 testdata.508946 2026-02-13T02:42:03.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj605 testdata.508946 2026-02-13T02:42:03.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj606 testdata.508946 2026-02-13T02:42:03.052 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.052 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj607 testdata.508946 2026-02-13T02:42:03.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj608 testdata.508946 2026-02-13T02:42:03.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj609 testdata.508946 2026-02-13T02:42:03.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj610 testdata.508946 2026-02-13T02:42:03.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj611 testdata.508946 2026-02-13T02:42:03.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj612 testdata.508946 2026-02-13T02:42:03.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj613 testdata.508946 2026-02-13T02:42:03.217 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.218 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj614 testdata.508946 2026-02-13T02:42:03.243 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.243 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj615 testdata.508946 2026-02-13T02:42:03.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj616 testdata.508946 2026-02-13T02:42:03.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj617 testdata.508946 2026-02-13T02:42:03.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj618 testdata.508946 2026-02-13T02:42:03.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj619 testdata.508946 2026-02-13T02:42:03.361 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.361 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj620 testdata.508946 2026-02-13T02:42:03.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj621 testdata.508946 2026-02-13T02:42:03.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj622 testdata.508946 2026-02-13T02:42:03.435 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.435 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj623 testdata.508946 2026-02-13T02:42:03.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj624 testdata.508946 2026-02-13T02:42:03.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj625 testdata.508946 2026-02-13T02:42:03.494 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj626 testdata.508946 2026-02-13T02:42:03.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj627 testdata.508946 2026-02-13T02:42:03.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj628 testdata.508946 2026-02-13T02:42:03.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj629 testdata.508946 2026-02-13T02:42:03.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj630 testdata.508946 2026-02-13T02:42:03.615 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.615 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj631 testdata.508946 2026-02-13T02:42:03.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj632 testdata.508946 2026-02-13T02:42:03.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj633 testdata.508946 2026-02-13T02:42:03.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj634 testdata.508946 2026-02-13T02:42:03.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj635 testdata.508946 2026-02-13T02:42:03.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj636 testdata.508946 2026-02-13T02:42:03.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj637 testdata.508946 2026-02-13T02:42:03.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.782 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj638 testdata.508946 2026-02-13T02:42:03.806 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj639 testdata.508946 2026-02-13T02:42:03.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj640 testdata.508946 2026-02-13T02:42:03.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj641 testdata.508946 2026-02-13T02:42:03.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj642 testdata.508946 2026-02-13T02:42:03.905 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.906 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj643 testdata.508946 2026-02-13T02:42:03.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj644 testdata.508946 2026-02-13T02:42:03.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj645 testdata.508946 2026-02-13T02:42:03.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj646 testdata.508946 2026-02-13T02:42:03.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:03.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj647 testdata.508946 2026-02-13T02:42:04.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj648 testdata.508946 2026-02-13T02:42:04.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj649 testdata.508946 2026-02-13T02:42:04.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj650 testdata.508946 2026-02-13T02:42:04.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj651 testdata.508946 2026-02-13T02:42:04.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj652 testdata.508946 2026-02-13T02:42:04.136 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.136 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj653 testdata.508946 2026-02-13T02:42:04.160 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj654 testdata.508946 2026-02-13T02:42:04.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj655 testdata.508946 2026-02-13T02:42:04.210 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.210 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj656 testdata.508946 2026-02-13T02:42:04.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj657 testdata.508946 2026-02-13T02:42:04.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj658 testdata.508946 2026-02-13T02:42:04.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj659 testdata.508946 2026-02-13T02:42:04.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj660 testdata.508946 2026-02-13T02:42:04.332 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj661 testdata.508946 2026-02-13T02:42:04.357 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.357 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj662 testdata.508946 2026-02-13T02:42:04.382 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.382 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj663 testdata.508946 2026-02-13T02:42:04.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj664 testdata.508946 2026-02-13T02:42:04.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj665 testdata.508946 2026-02-13T02:42:04.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj666 testdata.508946 2026-02-13T02:42:04.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj667 testdata.508946 2026-02-13T02:42:04.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj668 testdata.508946 2026-02-13T02:42:04.527 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.527 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj669 testdata.508946 2026-02-13T02:42:04.548 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.548 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj670 testdata.508946 2026-02-13T02:42:04.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj671 testdata.508946 2026-02-13T02:42:04.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.595 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj672 testdata.508946 2026-02-13T02:42:04.620 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.620 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj673 testdata.508946 2026-02-13T02:42:04.644 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.644 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj674 testdata.508946 2026-02-13T02:42:04.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj675 testdata.508946 2026-02-13T02:42:04.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.693 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj676 testdata.508946 2026-02-13T02:42:04.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj677 testdata.508946 2026-02-13T02:42:04.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj678 testdata.508946 2026-02-13T02:42:04.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj679 testdata.508946 2026-02-13T02:42:04.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.780 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj680 testdata.508946 2026-02-13T02:42:04.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj681 testdata.508946 2026-02-13T02:42:04.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj682 testdata.508946 2026-02-13T02:42:04.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj683 testdata.508946 2026-02-13T02:42:04.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj684 testdata.508946 2026-02-13T02:42:04.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj685 testdata.508946 2026-02-13T02:42:04.927 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.928 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj686 testdata.508946 2026-02-13T02:42:04.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj687 testdata.508946 2026-02-13T02:42:04.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:04.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj688 testdata.508946 2026-02-13T02:42:05.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj689 testdata.508946 2026-02-13T02:42:05.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj690 testdata.508946 2026-02-13T02:42:05.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj691 testdata.508946 2026-02-13T02:42:05.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj692 testdata.508946 2026-02-13T02:42:05.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj693 testdata.508946 2026-02-13T02:42:05.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj694 testdata.508946 2026-02-13T02:42:05.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.140 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj695 testdata.508946 2026-02-13T02:42:05.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj696 testdata.508946 2026-02-13T02:42:05.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.190 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj697 testdata.508946 2026-02-13T02:42:05.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj698 testdata.508946 2026-02-13T02:42:05.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj699 testdata.508946 2026-02-13T02:42:05.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj700 testdata.508946 2026-02-13T02:42:05.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.287 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj701 testdata.508946 2026-02-13T02:42:05.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.312 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj702 testdata.508946 2026-02-13T02:42:05.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj703 testdata.508946 2026-02-13T02:42:05.361 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.361 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj704 testdata.508946 2026-02-13T02:42:05.385 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.385 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj705 testdata.508946 2026-02-13T02:42:05.402 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.403 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj706 testdata.508946 2026-02-13T02:42:05.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj707 testdata.508946 2026-02-13T02:42:05.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj708 testdata.508946 2026-02-13T02:42:05.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj709 testdata.508946 2026-02-13T02:42:05.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj710 testdata.508946 2026-02-13T02:42:05.513 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.513 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj711 testdata.508946 2026-02-13T02:42:05.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj712 testdata.508946 2026-02-13T02:42:05.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj713 testdata.508946 2026-02-13T02:42:05.575 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.575 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj714 testdata.508946 2026-02-13T02:42:05.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.594 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj715 testdata.508946 2026-02-13T02:42:05.618 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.618 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj716 testdata.508946 2026-02-13T02:42:05.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj717 testdata.508946 2026-02-13T02:42:05.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj718 testdata.508946 2026-02-13T02:42:05.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.679 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj719 testdata.508946 2026-02-13T02:42:05.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj720 testdata.508946 2026-02-13T02:42:05.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj721 testdata.508946 2026-02-13T02:42:05.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj722 testdata.508946 2026-02-13T02:42:05.775 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.775 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj723 testdata.508946 2026-02-13T02:42:05.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj724 testdata.508946 2026-02-13T02:42:05.823 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj725 testdata.508946 2026-02-13T02:42:05.847 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.848 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj726 testdata.508946 2026-02-13T02:42:05.872 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj727 testdata.508946 2026-02-13T02:42:05.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj728 testdata.508946 2026-02-13T02:42:05.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj729 testdata.508946 2026-02-13T02:42:05.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj730 testdata.508946 2026-02-13T02:42:05.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj731 testdata.508946 2026-02-13T02:42:05.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:05.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj732 testdata.508946 2026-02-13T02:42:06.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj733 testdata.508946 2026-02-13T02:42:06.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj734 testdata.508946 2026-02-13T02:42:06.057 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.057 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj735 testdata.508946 2026-02-13T02:42:06.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.080 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj736 testdata.508946 2026-02-13T02:42:06.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj737 testdata.508946 2026-02-13T02:42:06.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj738 testdata.508946 2026-02-13T02:42:06.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.145 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj739 testdata.508946 2026-02-13T02:42:06.167 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.167 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj740 testdata.508946 2026-02-13T02:42:06.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj741 testdata.508946 2026-02-13T02:42:06.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj742 testdata.508946 2026-02-13T02:42:06.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj743 testdata.508946 2026-02-13T02:42:06.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.263 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj744 testdata.508946 2026-02-13T02:42:06.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj745 testdata.508946 2026-02-13T02:42:06.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj746 testdata.508946 2026-02-13T02:42:06.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj747 testdata.508946 2026-02-13T02:42:06.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj748 testdata.508946 2026-02-13T02:42:06.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj749 testdata.508946 2026-02-13T02:42:06.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj750 testdata.508946 2026-02-13T02:42:06.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj751 testdata.508946 2026-02-13T02:42:06.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj752 testdata.508946 2026-02-13T02:42:06.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj753 testdata.508946 2026-02-13T02:42:06.509 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.510 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj754 testdata.508946 2026-02-13T02:42:06.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj755 testdata.508946 2026-02-13T02:42:06.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj756 testdata.508946 2026-02-13T02:42:06.576 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.576 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj757 testdata.508946 2026-02-13T02:42:06.599 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.599 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj758 testdata.508946 2026-02-13T02:42:06.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj759 testdata.508946 2026-02-13T02:42:06.642 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.642 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj760 testdata.508946 2026-02-13T02:42:06.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj761 testdata.508946 2026-02-13T02:42:06.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj762 testdata.508946 2026-02-13T02:42:06.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj763 testdata.508946 2026-02-13T02:42:06.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj764 testdata.508946 2026-02-13T02:42:06.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj765 testdata.508946 2026-02-13T02:42:06.773 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.774 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj766 testdata.508946 2026-02-13T02:42:06.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj767 testdata.508946 2026-02-13T02:42:06.823 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.823 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj768 testdata.508946 2026-02-13T02:42:06.847 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.847 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj769 testdata.508946 2026-02-13T02:42:06.871 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.871 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj770 testdata.508946 2026-02-13T02:42:06.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj771 testdata.508946 2026-02-13T02:42:06.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj772 testdata.508946 2026-02-13T02:42:06.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj773 testdata.508946 2026-02-13T02:42:06.970 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.970 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj774 testdata.508946 2026-02-13T02:42:06.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:06.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj775 testdata.508946 2026-02-13T02:42:07.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj776 testdata.508946 2026-02-13T02:42:07.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj777 testdata.508946 2026-02-13T02:42:07.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj778 testdata.508946 2026-02-13T02:42:07.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj779 testdata.508946 2026-02-13T02:42:07.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj780 testdata.508946 2026-02-13T02:42:07.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj781 testdata.508946 2026-02-13T02:42:07.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj782 testdata.508946 2026-02-13T02:42:07.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj783 testdata.508946 2026-02-13T02:42:07.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj784 testdata.508946 2026-02-13T02:42:07.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj785 testdata.508946 2026-02-13T02:42:07.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj786 testdata.508946 2026-02-13T02:42:07.277 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.277 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj787 testdata.508946 2026-02-13T02:42:07.295 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.295 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj788 testdata.508946 2026-02-13T02:42:07.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj789 testdata.508946 2026-02-13T02:42:07.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj790 testdata.508946 2026-02-13T02:42:07.358 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.358 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj791 testdata.508946 2026-02-13T02:42:07.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj792 testdata.508946 2026-02-13T02:42:07.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj793 testdata.508946 2026-02-13T02:42:07.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj794 testdata.508946 2026-02-13T02:42:07.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj795 testdata.508946 2026-02-13T02:42:07.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj796 testdata.508946 2026-02-13T02:42:07.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj797 testdata.508946 2026-02-13T02:42:07.510 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.510 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj798 testdata.508946 2026-02-13T02:42:07.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj799 testdata.508946 2026-02-13T02:42:07.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj800 testdata.508946 2026-02-13T02:42:07.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj801 testdata.508946 2026-02-13T02:42:07.601 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.601 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj802 testdata.508946 2026-02-13T02:42:07.624 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.624 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj803 testdata.508946 2026-02-13T02:42:07.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj804 testdata.508946 2026-02-13T02:42:07.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj805 testdata.508946 2026-02-13T02:42:07.692 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.692 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj806 testdata.508946 2026-02-13T02:42:07.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj807 testdata.508946 2026-02-13T02:42:07.735 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.735 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj808 testdata.508946 2026-02-13T02:42:07.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj809 testdata.508946 2026-02-13T02:42:07.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj810 testdata.508946 2026-02-13T02:42:07.807 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj811 testdata.508946 2026-02-13T02:42:07.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj812 testdata.508946 2026-02-13T02:42:07.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj813 testdata.508946 2026-02-13T02:42:07.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj814 testdata.508946 2026-02-13T02:42:07.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj815 testdata.508946 2026-02-13T02:42:07.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj816 testdata.508946 2026-02-13T02:42:07.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj817 testdata.508946 2026-02-13T02:42:07.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj818 testdata.508946 2026-02-13T02:42:07.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:07.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj819 testdata.508946 2026-02-13T02:42:08.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj820 testdata.508946 2026-02-13T02:42:08.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj821 testdata.508946 2026-02-13T02:42:08.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj822 testdata.508946 2026-02-13T02:42:08.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj823 testdata.508946 2026-02-13T02:42:08.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj824 testdata.508946 2026-02-13T02:42:08.136 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.136 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj825 testdata.508946 2026-02-13T02:42:08.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj826 testdata.508946 2026-02-13T02:42:08.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj827 testdata.508946 2026-02-13T02:42:08.210 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.210 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj828 testdata.508946 2026-02-13T02:42:08.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj829 testdata.508946 2026-02-13T02:42:08.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj830 testdata.508946 2026-02-13T02:42:08.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj831 testdata.508946 2026-02-13T02:42:08.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj832 testdata.508946 2026-02-13T02:42:08.331 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.331 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj833 testdata.508946 2026-02-13T02:42:08.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj834 testdata.508946 2026-02-13T02:42:08.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.381 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj835 testdata.508946 2026-02-13T02:42:08.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj836 testdata.508946 2026-02-13T02:42:08.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj837 testdata.508946 2026-02-13T02:42:08.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj838 testdata.508946 2026-02-13T02:42:08.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj839 testdata.508946 2026-02-13T02:42:08.497 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.498 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj840 testdata.508946 2026-02-13T02:42:08.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj841 testdata.508946 2026-02-13T02:42:08.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj842 testdata.508946 2026-02-13T02:42:08.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj843 testdata.508946 2026-02-13T02:42:08.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj844 testdata.508946 2026-02-13T02:42:08.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj845 testdata.508946 2026-02-13T02:42:08.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj846 testdata.508946 2026-02-13T02:42:08.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj847 testdata.508946 2026-02-13T02:42:08.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj848 testdata.508946 2026-02-13T02:42:08.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj849 testdata.508946 2026-02-13T02:42:08.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj850 testdata.508946 2026-02-13T02:42:08.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj851 testdata.508946 2026-02-13T02:42:08.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj852 testdata.508946 2026-02-13T02:42:08.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj853 testdata.508946 2026-02-13T02:42:08.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj854 testdata.508946 2026-02-13T02:42:08.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj855 testdata.508946 2026-02-13T02:42:08.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj856 testdata.508946 2026-02-13T02:42:08.899 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.899 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj857 testdata.508946 2026-02-13T02:42:08.924 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.924 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj858 testdata.508946 2026-02-13T02:42:08.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj859 testdata.508946 2026-02-13T02:42:08.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj860 testdata.508946 2026-02-13T02:42:08.988 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:08.988 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj861 testdata.508946 2026-02-13T02:42:09.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj862 testdata.508946 2026-02-13T02:42:09.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj863 testdata.508946 2026-02-13T02:42:09.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj864 testdata.508946 2026-02-13T02:42:09.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj865 testdata.508946 2026-02-13T02:42:09.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj866 testdata.508946 2026-02-13T02:42:09.136 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.136 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj867 testdata.508946 2026-02-13T02:42:09.160 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.160 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj868 testdata.508946 2026-02-13T02:42:09.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj869 testdata.508946 2026-02-13T02:42:09.210 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.210 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj870 testdata.508946 2026-02-13T02:42:09.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.228 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj871 testdata.508946 2026-02-13T02:42:09.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj872 testdata.508946 2026-02-13T02:42:09.267 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.267 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj873 testdata.508946 2026-02-13T02:42:09.291 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.291 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj874 testdata.508946 2026-02-13T02:42:09.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj875 testdata.508946 2026-02-13T02:42:09.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj876 testdata.508946 2026-02-13T02:42:09.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj877 testdata.508946 2026-02-13T02:42:09.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj878 testdata.508946 2026-02-13T02:42:09.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj879 testdata.508946 2026-02-13T02:42:09.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj880 testdata.508946 2026-02-13T02:42:09.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.461 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj881 testdata.508946 2026-02-13T02:42:09.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj882 testdata.508946 2026-02-13T02:42:09.511 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.511 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj883 testdata.508946 2026-02-13T02:42:09.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj884 testdata.508946 2026-02-13T02:42:09.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj885 testdata.508946 2026-02-13T02:42:09.584 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.584 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj886 testdata.508946 2026-02-13T02:42:09.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj887 testdata.508946 2026-02-13T02:42:09.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj888 testdata.508946 2026-02-13T02:42:09.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj889 testdata.508946 2026-02-13T02:42:09.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj890 testdata.508946 2026-02-13T02:42:09.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj891 testdata.508946 2026-02-13T02:42:09.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj892 testdata.508946 2026-02-13T02:42:09.737 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj893 testdata.508946 2026-02-13T02:42:09.762 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.762 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj894 testdata.508946 2026-02-13T02:42:09.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj895 testdata.508946 2026-02-13T02:42:09.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj896 testdata.508946 2026-02-13T02:42:09.836 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.836 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj897 testdata.508946 2026-02-13T02:42:09.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj898 testdata.508946 2026-02-13T02:42:09.886 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.886 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj899 testdata.508946 2026-02-13T02:42:09.910 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.910 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj900 testdata.508946 2026-02-13T02:42:09.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj901 testdata.508946 2026-02-13T02:42:09.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj902 testdata.508946 2026-02-13T02:42:09.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj903 testdata.508946 2026-02-13T02:42:09.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:09.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj904 testdata.508946 2026-02-13T02:42:10.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj905 testdata.508946 2026-02-13T02:42:10.037 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.037 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj906 testdata.508946 2026-02-13T02:42:10.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj907 testdata.508946 2026-02-13T02:42:10.086 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj908 testdata.508946 2026-02-13T02:42:10.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj909 testdata.508946 2026-02-13T02:42:10.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj910 testdata.508946 2026-02-13T02:42:10.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj911 testdata.508946 2026-02-13T02:42:10.182 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.182 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj912 testdata.508946 2026-02-13T02:42:10.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj913 testdata.508946 2026-02-13T02:42:10.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj914 testdata.508946 2026-02-13T02:42:10.254 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.254 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj915 testdata.508946 2026-02-13T02:42:10.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj916 testdata.508946 2026-02-13T02:42:10.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj917 testdata.508946 2026-02-13T02:42:10.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj918 testdata.508946 2026-02-13T02:42:10.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.347 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj919 testdata.508946 2026-02-13T02:42:10.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj920 testdata.508946 2026-02-13T02:42:10.395 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.395 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj921 testdata.508946 2026-02-13T02:42:10.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj922 testdata.508946 2026-02-13T02:42:10.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj923 testdata.508946 2026-02-13T02:42:10.469 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.469 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj924 testdata.508946 2026-02-13T02:42:10.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj925 testdata.508946 2026-02-13T02:42:10.507 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.508 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj926 testdata.508946 2026-02-13T02:42:10.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.532 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj927 testdata.508946 2026-02-13T02:42:10.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj928 testdata.508946 2026-02-13T02:42:10.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj929 testdata.508946 2026-02-13T02:42:10.597 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.597 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj930 testdata.508946 2026-02-13T02:42:10.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj931 testdata.508946 2026-02-13T02:42:10.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj932 testdata.508946 2026-02-13T02:42:10.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj933 testdata.508946 2026-02-13T02:42:10.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj934 testdata.508946 2026-02-13T02:42:10.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj935 testdata.508946 2026-02-13T02:42:10.735 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.735 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj936 testdata.508946 2026-02-13T02:42:10.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj937 testdata.508946 2026-02-13T02:42:10.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj938 testdata.508946 2026-02-13T02:42:10.801 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj939 testdata.508946 2026-02-13T02:42:10.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj940 testdata.508946 2026-02-13T02:42:10.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj941 testdata.508946 2026-02-13T02:42:10.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj942 testdata.508946 2026-02-13T02:42:10.892 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.893 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj943 testdata.508946 2026-02-13T02:42:10.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj944 testdata.508946 2026-02-13T02:42:10.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj945 testdata.508946 2026-02-13T02:42:10.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj946 testdata.508946 2026-02-13T02:42:10.988 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:10.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj947 testdata.508946 2026-02-13T02:42:11.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj948 testdata.508946 2026-02-13T02:42:11.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj949 testdata.508946 2026-02-13T02:42:11.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj950 testdata.508946 2026-02-13T02:42:11.086 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj951 testdata.508946 2026-02-13T02:42:11.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj952 testdata.508946 2026-02-13T02:42:11.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj953 testdata.508946 2026-02-13T02:42:11.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj954 testdata.508946 2026-02-13T02:42:11.184 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.184 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj955 testdata.508946 2026-02-13T02:42:11.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.210 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj956 testdata.508946 2026-02-13T02:42:11.232 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.232 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj957 testdata.508946 2026-02-13T02:42:11.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj958 testdata.508946 2026-02-13T02:42:11.280 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.280 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj959 testdata.508946 2026-02-13T02:42:11.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj960 testdata.508946 2026-02-13T02:42:11.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj961 testdata.508946 2026-02-13T02:42:11.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj962 testdata.508946 2026-02-13T02:42:11.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj963 testdata.508946 2026-02-13T02:42:11.403 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.403 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj964 testdata.508946 2026-02-13T02:42:11.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj965 testdata.508946 2026-02-13T02:42:11.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj966 testdata.508946 2026-02-13T02:42:11.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj967 testdata.508946 2026-02-13T02:42:11.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj968 testdata.508946 2026-02-13T02:42:11.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj969 testdata.508946 2026-02-13T02:42:11.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj970 testdata.508946 2026-02-13T02:42:11.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj971 testdata.508946 2026-02-13T02:42:11.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj972 testdata.508946 2026-02-13T02:42:11.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj973 testdata.508946 2026-02-13T02:42:11.642 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.642 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj974 testdata.508946 2026-02-13T02:42:11.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj975 testdata.508946 2026-02-13T02:42:11.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj976 testdata.508946 2026-02-13T02:42:11.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj977 testdata.508946 2026-02-13T02:42:11.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj978 testdata.508946 2026-02-13T02:42:11.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj979 testdata.508946 2026-02-13T02:42:11.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj980 testdata.508946 2026-02-13T02:42:11.801 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.801 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj981 testdata.508946 2026-02-13T02:42:11.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj982 testdata.508946 2026-02-13T02:42:11.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj983 testdata.508946 2026-02-13T02:42:11.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj984 testdata.508946 2026-02-13T02:42:11.892 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.893 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj985 testdata.508946 2026-02-13T02:42:11.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj986 testdata.508946 2026-02-13T02:42:11.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj987 testdata.508946 2026-02-13T02:42:11.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj988 testdata.508946 2026-02-13T02:42:11.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:11.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj989 testdata.508946 2026-02-13T02:42:12.006 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:12.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj990 testdata.508946 2026-02-13T02:42:12.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:12.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj991 testdata.508946 2026-02-13T02:42:12.051 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:12.051 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj992 testdata.508946 2026-02-13T02:42:12.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:12.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj993 testdata.508946 2026-02-13T02:42:12.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:12.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj994 testdata.508946 2026-02-13T02:42:12.125 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:12.125 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj995 testdata.508946 2026-02-13T02:42:12.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:12.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj996 testdata.508946 2026-02-13T02:42:12.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:12.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj997 testdata.508946 2026-02-13T02:42:12.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:12.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj998 testdata.508946 2026-02-13T02:42:12.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:12.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj999 testdata.508946 2026-02-13T02:42:12.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-13T02:42:12.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj1000 testdata.508946 2026-02-13T02:42:12.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:334: _scrub_abort: rm -f testdata.508946 2026-02-13T02:42:12.262 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:336: _scrub_abort: get_primary test obj1 2026-02-13T02:42:12.263 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2026-02-13T02:42:12.263 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-13T02:42:12.263 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2026-02-13T02:42:12.263 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T02:42:12.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:336: _scrub_abort: local primary=1 2026-02-13T02:42:12.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: local pgid=1.0 2026-02-13T02:42:12.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: ceph tell 1.0 schedule-scrub 2026-02-13T02:42:12.487 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:42:12.487 INFO:tasks.workunit.client.0.trial112.stdout: "deep": false, 2026-02-13T02:42:12.487 INFO:tasks.workunit.client.0.trial112.stdout: "must": false, 2026-02-13T02:42:12.487 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "2026-02-11T02:42:12.483336+0000" 2026-02-13T02:42:12.487 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:42:12.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:342: _scrub_abort: set -o pipefail 2026-02-13T02:42:12.495 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:343: _scrub_abort: found=no 2026-02-13T02:42:12.495 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:344: _scrub_abort: seq 0 200 2026-02-13T02:42:12.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:344: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:42:12.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:346: _scrub_abort: flush_pg_stats 2026-02-13T02:42:12.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:42:12.497 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:42:12.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:42:12.655 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:42:12.655 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:42:12.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:42:12.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:12.656 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:42:12.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836515 2026-02-13T02:42:12.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836515 2026-02-13T02:42:12.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836515' 2026-02-13T02:42:12.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:12.728 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:42:12.799 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705697 2026-02-13T02:42:12.799 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705697 2026-02-13T02:42:12.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836515 1-38654705697' 2026-02-13T02:42:12.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:12.800 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:42:12.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574878 2026-02-13T02:42:12.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574878 2026-02-13T02:42:12.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836515 1-38654705697 2-55834574878' 2026-02-13T02:42:12.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:12.874 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836515 2026-02-13T02:42:12.874 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:12.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:42:12.876 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836515 2026-02-13T02:42:12.876 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:12.877 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836515 2026-02-13T02:42:12.877 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836515' 2026-02-13T02:42:12.878 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836515 2026-02-13T02:42:12.878 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:42:13.047 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836513 -lt 21474836515 2026-02-13T02:42:13.048 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:42:14.049 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:42:14.050 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:42:14.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836516 -lt 21474836515 2026-02-13T02:42:14.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:14.209 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705697 2026-02-13T02:42:14.209 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:14.210 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:42:14.211 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705697 2026-02-13T02:42:14.211 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:14.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705697 2026-02-13T02:42:14.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705697' 2026-02-13T02:42:14.213 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705697 2026-02-13T02:42:14.214 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:42:14.385 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705697 -lt 38654705697 2026-02-13T02:42:14.385 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:14.385 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574878 2026-02-13T02:42:14.386 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:14.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:42:14.387 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574878 2026-02-13T02:42:14.388 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:14.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574878 2026-02-13T02:42:14.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574878' 2026-02-13T02:42:14.389 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574878 2026-02-13T02:42:14.390 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:42:14.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574879 -lt 55834574878 2026-02-13T02:42:14.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:347: _scrub_abort: ceph pg dump pgs 2026-02-13T02:42:14.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:347: _scrub_abort: grep '^1.0' 2026-02-13T02:42:14.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:347: _scrub_abort: grep -q scrubbing 2026-02-13T02:42:14.697 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:42:14.707 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:349: _scrub_abort: found=yes 2026-02-13T02:42:14.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:351: _scrub_abort: break 2026-02-13T02:42:14.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:354: _scrub_abort: set +o pipefail 2026-02-13T02:42:14.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:356: _scrub_abort: test yes = no 2026-02-13T02:42:14.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:362: _scrub_abort: ceph osd set noscrub 2026-02-13T02:42:14.915 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is set 2026-02-13T02:42:14.927 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:363: _scrub_abort: '[' scrub = deep-scrub ']' 2026-02-13T02:42:14.927 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:369: _scrub_abort: set -o pipefail 2026-02-13T02:42:14.927 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: seq 0 200 2026-02-13T02:42:14.928 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:42:14.928 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:42:14.928 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:42:14.929 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:42:15.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:42:15.088 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:42:15.089 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:42:15.089 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:42:15.089 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:15.089 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:42:15.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836519 2026-02-13T02:42:15.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836519 2026-02-13T02:42:15.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836519' 2026-02-13T02:42:15.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:15.162 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:42:15.232 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705700 2026-02-13T02:42:15.232 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705700 2026-02-13T02:42:15.232 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836519 1-38654705700' 2026-02-13T02:42:15.232 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:15.233 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:42:15.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574881 2026-02-13T02:42:15.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574881 2026-02-13T02:42:15.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836519 1-38654705700 2-55834574881' 2026-02-13T02:42:15.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:15.306 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836519 2026-02-13T02:42:15.306 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:15.307 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:42:15.308 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836519 2026-02-13T02:42:15.308 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:15.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836519 2026-02-13T02:42:15.310 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836519 2026-02-13T02:42:15.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836519' 2026-02-13T02:42:15.310 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:42:15.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836516 -lt 21474836519 2026-02-13T02:42:15.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:42:16.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:42:16.477 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:42:16.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836519 -lt 21474836519 2026-02-13T02:42:16.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:16.634 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705700 2026-02-13T02:42:16.634 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:16.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:42:16.636 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705700 2026-02-13T02:42:16.636 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:16.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705700 2026-02-13T02:42:16.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705700' 2026-02-13T02:42:16.638 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705700 2026-02-13T02:42:16.638 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:42:16.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705700 -lt 38654705700 2026-02-13T02:42:16.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:16.795 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574881 2026-02-13T02:42:16.795 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:16.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:42:16.797 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574881 2026-02-13T02:42:16.797 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:16.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574881 2026-02-13T02:42:16.799 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574881 2026-02-13T02:42:16.799 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574881' 2026-02-13T02:42:16.799 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:42:16.963 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574882 -lt 55834574881 2026-02-13T02:42:16.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:42:16.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:42:16.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:42:17.111 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:42:17.121 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:42:17.121 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:42:17.121 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:42:17.121 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:42:17.121 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:42:17.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:42:17.273 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:42:17.273 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:42:17.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:42:17.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:17.274 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:42:17.348 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836522 2026-02-13T02:42:17.348 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836522 2026-02-13T02:42:17.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836522' 2026-02-13T02:42:17.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:17.349 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:42:17.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705703 2026-02-13T02:42:17.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705703 2026-02-13T02:42:17.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836522 1-38654705703' 2026-02-13T02:42:17.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:17.421 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:42:17.498 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574885 2026-02-13T02:42:17.498 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574885 2026-02-13T02:42:17.498 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836522 1-38654705703 2-55834574885' 2026-02-13T02:42:17.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:17.499 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836522 2026-02-13T02:42:17.499 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:17.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:42:17.501 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836522 2026-02-13T02:42:17.501 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:17.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836522 2026-02-13T02:42:17.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836522' 2026-02-13T02:42:17.503 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836522 2026-02-13T02:42:17.503 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:42:17.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836519 -lt 21474836522 2026-02-13T02:42:17.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:42:18.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:42:18.669 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:42:18.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836522 -lt 21474836522 2026-02-13T02:42:18.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:18.827 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705703 2026-02-13T02:42:18.827 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:18.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:42:18.829 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705703 2026-02-13T02:42:18.829 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:18.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705703 2026-02-13T02:42:18.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705703' 2026-02-13T02:42:18.831 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705703 2026-02-13T02:42:18.831 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:42:18.998 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705703 -lt 38654705703 2026-02-13T02:42:18.998 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:18.999 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574885 2026-02-13T02:42:18.999 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:19.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:42:19.001 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574885 2026-02-13T02:42:19.001 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:19.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574885 2026-02-13T02:42:19.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574885' 2026-02-13T02:42:19.003 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574885 2026-02-13T02:42:19.003 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:42:19.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574885 -lt 55834574885 2026-02-13T02:42:19.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:42:19.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:42:19.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:42:19.313 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:42:19.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:42:19.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:42:19.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:42:19.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:42:19.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:42:19.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:42:19.481 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:42:19.481 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:42:19.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:42:19.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:19.482 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:42:19.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836525 2026-02-13T02:42:19.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836525 2026-02-13T02:42:19.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836525' 2026-02-13T02:42:19.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:19.554 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:42:19.626 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705706 2026-02-13T02:42:19.626 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705706 2026-02-13T02:42:19.626 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836525 1-38654705706' 2026-02-13T02:42:19.626 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:19.626 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:42:19.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574888 2026-02-13T02:42:19.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574888 2026-02-13T02:42:19.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836525 1-38654705706 2-55834574888' 2026-02-13T02:42:19.696 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:19.697 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836525 2026-02-13T02:42:19.697 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:19.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:42:19.699 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836525 2026-02-13T02:42:19.699 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:19.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836525 2026-02-13T02:42:19.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836525' 2026-02-13T02:42:19.701 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836525 2026-02-13T02:42:19.701 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:42:19.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836525 -lt 21474836525 2026-02-13T02:42:19.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:19.863 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705706 2026-02-13T02:42:19.863 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:19.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:42:19.865 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705706 2026-02-13T02:42:19.865 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:19.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705706 2026-02-13T02:42:19.867 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705706 2026-02-13T02:42:19.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705706' 2026-02-13T02:42:19.868 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:42:20.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705706 -lt 38654705706 2026-02-13T02:42:20.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:20.029 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574888 2026-02-13T02:42:20.029 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:20.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:42:20.030 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574888 2026-02-13T02:42:20.031 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:20.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574888 2026-02-13T02:42:20.032 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574888 2026-02-13T02:42:20.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574888' 2026-02-13T02:42:20.033 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:42:20.199 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574887 -lt 55834574888 2026-02-13T02:42:20.199 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:42:21.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:42:21.201 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:42:21.361 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574887 -lt 55834574888 2026-02-13T02:42:21.361 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:42:22.363 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 499 -eq 0 ']' 2026-02-13T02:42:22.363 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:42:22.523 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574890 -lt 55834574888 2026-02-13T02:42:22.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:42:22.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:42:22.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:42:22.675 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:42:22.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:42:22.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:42:22.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:42:22.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:42:22.686 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:42:22.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:42:22.846 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:42:22.846 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:42:22.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:42:22.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:22.847 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:42:22.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836530 2026-02-13T02:42:22.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836530 2026-02-13T02:42:22.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836530' 2026-02-13T02:42:22.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:22.918 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:42:22.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705711 2026-02-13T02:42:22.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705711 2026-02-13T02:42:22.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836530 1-38654705711' 2026-02-13T02:42:22.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:22.992 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:42:23.066 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574892 2026-02-13T02:42:23.066 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574892 2026-02-13T02:42:23.066 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836530 1-38654705711 2-55834574892' 2026-02-13T02:42:23.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:23.067 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836530 2026-02-13T02:42:23.067 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:23.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:42:23.069 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836530 2026-02-13T02:42:23.069 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:23.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836530 2026-02-13T02:42:23.070 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836530 2026-02-13T02:42:23.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836530' 2026-02-13T02:42:23.071 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:42:23.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836527 -lt 21474836530 2026-02-13T02:42:23.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:42:24.232 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:42:24.232 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:42:24.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836530 -lt 21474836530 2026-02-13T02:42:24.393 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:24.393 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705711 2026-02-13T02:42:24.393 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:24.395 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:42:24.396 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705711 2026-02-13T02:42:24.396 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:24.397 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705711 2026-02-13T02:42:24.397 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705711 2026-02-13T02:42:24.398 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705711' 2026-02-13T02:42:24.398 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:42:24.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705711 -lt 38654705711 2026-02-13T02:42:24.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:24.567 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574892 2026-02-13T02:42:24.568 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:24.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:42:24.570 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574892 2026-02-13T02:42:24.570 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:24.571 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574892 2026-02-13T02:42:24.572 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574892' 2026-02-13T02:42:24.572 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574892 2026-02-13T02:42:24.572 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:42:24.741 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574893 -lt 55834574892 2026-02-13T02:42:24.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:42:24.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:42:24.742 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:42:24.889 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:42:24.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:42:24.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:42:24.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:42:24.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:42:24.900 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:42:25.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:42:25.060 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:42:25.061 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:42:25.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:42:25.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:25.061 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:42:25.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836533 2026-02-13T02:42:25.136 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836533 2026-02-13T02:42:25.136 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836533' 2026-02-13T02:42:25.136 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:25.136 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:42:25.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705714 2026-02-13T02:42:25.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705714 2026-02-13T02:42:25.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836533 1-38654705714' 2026-02-13T02:42:25.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:25.209 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:42:25.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574895 2026-02-13T02:42:25.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574895 2026-02-13T02:42:25.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836533 1-38654705714 2-55834574895' 2026-02-13T02:42:25.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:25.285 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836533 2026-02-13T02:42:25.285 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:25.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:42:25.287 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836533 2026-02-13T02:42:25.287 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:25.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836533 2026-02-13T02:42:25.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836533' 2026-02-13T02:42:25.289 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836533 2026-02-13T02:42:25.289 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:42:25.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836530 -lt 21474836533 2026-02-13T02:42:25.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:42:26.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:42:26.451 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:42:26.615 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836533 -lt 21474836533 2026-02-13T02:42:26.615 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:26.616 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705714 2026-02-13T02:42:26.616 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:26.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:42:26.618 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705714 2026-02-13T02:42:26.618 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:26.619 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705714 2026-02-13T02:42:26.620 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705714 2026-02-13T02:42:26.620 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705714' 2026-02-13T02:42:26.620 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:42:26.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705714 -lt 38654705714 2026-02-13T02:42:26.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:26.792 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574895 2026-02-13T02:42:26.793 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:26.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:42:26.795 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574895 2026-02-13T02:42:26.795 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:26.796 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574895 2026-02-13T02:42:26.796 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574895 2026-02-13T02:42:26.797 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574895' 2026-02-13T02:42:26.797 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:42:26.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574896 -lt 55834574895 2026-02-13T02:42:26.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:42:26.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:42:26.967 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:42:27.122 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:42:27.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:42:27.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:42:27.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:42:27.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:42:27.133 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:42:27.291 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:42:27.291 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:42:27.291 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:42:27.291 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:42:27.291 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:27.292 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:42:27.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836536 2026-02-13T02:42:27.365 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836536 2026-02-13T02:42:27.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836536' 2026-02-13T02:42:27.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:27.366 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:42:27.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705717 2026-02-13T02:42:27.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705717 2026-02-13T02:42:27.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836536 1-38654705717' 2026-02-13T02:42:27.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:27.439 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:42:27.509 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574899 2026-02-13T02:42:27.509 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574899 2026-02-13T02:42:27.509 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836536 1-38654705717 2-55834574899' 2026-02-13T02:42:27.509 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:27.509 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836536 2026-02-13T02:42:27.510 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:27.511 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:42:27.511 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836536 2026-02-13T02:42:27.512 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:27.513 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836536 2026-02-13T02:42:27.513 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836536 2026-02-13T02:42:27.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836536' 2026-02-13T02:42:27.514 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:42:27.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836533 -lt 21474836536 2026-02-13T02:42:27.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:42:28.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:42:28.684 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:42:28.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836536 -lt 21474836536 2026-02-13T02:42:28.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:28.845 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705717 2026-02-13T02:42:28.845 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:28.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:42:28.847 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705717 2026-02-13T02:42:28.847 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:28.848 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705717 2026-02-13T02:42:28.849 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705717 2026-02-13T02:42:28.849 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705717' 2026-02-13T02:42:28.849 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:42:29.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705717 -lt 38654705717 2026-02-13T02:42:29.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:29.020 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574899 2026-02-13T02:42:29.020 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:29.021 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:42:29.022 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574899 2026-02-13T02:42:29.022 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:29.023 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574899 2026-02-13T02:42:29.023 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574899 2026-02-13T02:42:29.024 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574899' 2026-02-13T02:42:29.024 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:42:29.184 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574899 -lt 55834574899 2026-02-13T02:42:29.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:42:29.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:42:29.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:42:29.336 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:42:29.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:42:29.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:42:29.347 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:42:29.347 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:42:29.347 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:42:29.509 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:42:29.509 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:42:29.509 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:42:29.510 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:42:29.510 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:29.510 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:42:29.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836539 2026-02-13T02:42:29.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836539 2026-02-13T02:42:29.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836539' 2026-02-13T02:42:29.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:29.584 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:42:29.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705720 2026-02-13T02:42:29.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705720 2026-02-13T02:42:29.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836539 1-38654705720' 2026-02-13T02:42:29.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:29.654 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:42:29.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574902 2026-02-13T02:42:29.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574902 2026-02-13T02:42:29.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836539 1-38654705720 2-55834574902' 2026-02-13T02:42:29.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:29.727 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836539 2026-02-13T02:42:29.727 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:29.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:42:29.729 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836539 2026-02-13T02:42:29.729 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:29.730 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836539 2026-02-13T02:42:29.730 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836539 2026-02-13T02:42:29.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836539' 2026-02-13T02:42:29.731 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:42:29.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836539 -lt 21474836539 2026-02-13T02:42:29.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:29.889 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705720 2026-02-13T02:42:29.889 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:29.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:42:29.891 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705720 2026-02-13T02:42:29.892 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:29.893 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705720 2026-02-13T02:42:29.893 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705720' 2026-02-13T02:42:29.893 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705720 2026-02-13T02:42:29.894 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:42:30.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705720 -lt 38654705720 2026-02-13T02:42:30.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:30.064 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574902 2026-02-13T02:42:30.064 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:30.065 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:42:30.066 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574902 2026-02-13T02:42:30.066 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:30.067 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574902 2026-02-13T02:42:30.067 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574902 2026-02-13T02:42:30.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574902' 2026-02-13T02:42:30.068 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:42:30.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574901 -lt 55834574902 2026-02-13T02:42:30.239 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:42:31.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:42:31.240 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:42:31.395 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574901 -lt 55834574902 2026-02-13T02:42:31.395 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:42:32.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 499 -eq 0 ']' 2026-02-13T02:42:32.396 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:42:32.556 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574904 -lt 55834574902 2026-02-13T02:42:32.557 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:42:32.557 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:42:32.557 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:42:32.706 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:42:32.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:42:32.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:42:32.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:42:32.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:42:32.716 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:42:32.871 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:42:32.871 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:42:32.871 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:42:32.871 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:42:32.872 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:32.872 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:42:32.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836544 2026-02-13T02:42:32.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836544 2026-02-13T02:42:32.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836544' 2026-02-13T02:42:32.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:32.946 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:42:33.017 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705725 2026-02-13T02:42:33.017 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705725 2026-02-13T02:42:33.017 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836544 1-38654705725' 2026-02-13T02:42:33.017 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:33.017 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:42:33.089 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574906 2026-02-13T02:42:33.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574906 2026-02-13T02:42:33.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836544 1-38654705725 2-55834574906' 2026-02-13T02:42:33.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:33.090 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836544 2026-02-13T02:42:33.090 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:33.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:42:33.092 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836544 2026-02-13T02:42:33.093 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:33.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836544 2026-02-13T02:42:33.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836544' 2026-02-13T02:42:33.095 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836544 2026-02-13T02:42:33.095 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:42:33.258 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836541 -lt 21474836544 2026-02-13T02:42:33.258 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:42:34.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:42:34.260 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:42:34.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836544 -lt 21474836544 2026-02-13T02:42:34.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:34.424 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705725 2026-02-13T02:42:34.424 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:34.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:42:34.426 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705725 2026-02-13T02:42:34.426 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:34.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705725 2026-02-13T02:42:34.428 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705725 2026-02-13T02:42:34.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705725' 2026-02-13T02:42:34.429 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:42:34.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705725 -lt 38654705725 2026-02-13T02:42:34.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:34.589 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574906 2026-02-13T02:42:34.589 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:34.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:42:34.591 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574906 2026-02-13T02:42:34.591 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:34.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574906 2026-02-13T02:42:34.593 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574906 2026-02-13T02:42:34.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574906' 2026-02-13T02:42:34.593 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:42:34.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574907 -lt 55834574906 2026-02-13T02:42:34.766 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:42:34.766 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:42:34.766 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:42:34.913 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:42:34.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:42:34.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:42:34.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:42:34.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:42:34.923 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:42:35.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:42:35.085 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:42:35.085 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:42:35.086 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:42:35.086 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:35.086 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:42:35.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836547 2026-02-13T02:42:35.156 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836547 2026-02-13T02:42:35.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836547' 2026-02-13T02:42:35.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:35.157 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:42:35.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705728 2026-02-13T02:42:35.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705728 2026-02-13T02:42:35.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836547 1-38654705728' 2026-02-13T02:42:35.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:35.232 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:42:35.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574909 2026-02-13T02:42:35.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574909 2026-02-13T02:42:35.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836547 1-38654705728 2-55834574909' 2026-02-13T02:42:35.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:35.304 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836547 2026-02-13T02:42:35.304 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:35.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:42:35.306 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836547 2026-02-13T02:42:35.306 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:35.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836547 2026-02-13T02:42:35.308 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836547 2026-02-13T02:42:35.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836547' 2026-02-13T02:42:35.309 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:42:35.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836544 -lt 21474836547 2026-02-13T02:42:35.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:42:36.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:42:36.478 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:42:36.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836547 -lt 21474836547 2026-02-13T02:42:36.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:36.639 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705728 2026-02-13T02:42:36.639 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:36.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:42:36.641 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705728 2026-02-13T02:42:36.641 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:36.643 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705728 2026-02-13T02:42:36.643 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705728' 2026-02-13T02:42:36.643 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705728 2026-02-13T02:42:36.644 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:42:36.801 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705728 -lt 38654705728 2026-02-13T02:42:36.801 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:36.802 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574909 2026-02-13T02:42:36.802 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:36.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:42:36.804 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574909 2026-02-13T02:42:36.804 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:36.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574909 2026-02-13T02:42:36.805 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574909 2026-02-13T02:42:36.805 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574909' 2026-02-13T02:42:36.806 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:42:36.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574910 -lt 55834574909 2026-02-13T02:42:36.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:42:36.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:42:36.977 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:42:37.123 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:42:37.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:42:37.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:42:37.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:42:37.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:42:37.133 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:42:37.291 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:42:37.291 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:42:37.292 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:42:37.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:42:37.292 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:37.292 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:42:37.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836550 2026-02-13T02:42:37.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836550 2026-02-13T02:42:37.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836550' 2026-02-13T02:42:37.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:37.367 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:42:37.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705731 2026-02-13T02:42:37.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705731 2026-02-13T02:42:37.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836550 1-38654705731' 2026-02-13T02:42:37.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:37.440 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:42:37.509 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574913 2026-02-13T02:42:37.509 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574913 2026-02-13T02:42:37.509 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836550 1-38654705731 2-55834574913' 2026-02-13T02:42:37.509 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:37.510 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836550 2026-02-13T02:42:37.510 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:37.511 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:42:37.512 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836550 2026-02-13T02:42:37.512 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:37.513 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836550 2026-02-13T02:42:37.513 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836550' 2026-02-13T02:42:37.514 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836550 2026-02-13T02:42:37.514 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:42:37.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836547 -lt 21474836550 2026-02-13T02:42:37.684 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:42:38.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:42:38.685 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:42:38.848 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836550 -lt 21474836550 2026-02-13T02:42:38.848 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:38.849 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705731 2026-02-13T02:42:38.849 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:38.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:42:38.851 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705731 2026-02-13T02:42:38.851 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:38.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705731 2026-02-13T02:42:38.853 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705731 2026-02-13T02:42:38.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705731' 2026-02-13T02:42:38.854 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:42:39.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705731 -lt 38654705731 2026-02-13T02:42:39.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:39.019 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574913 2026-02-13T02:42:39.019 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:39.021 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:42:39.021 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574913 2026-02-13T02:42:39.022 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:39.023 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574913 2026-02-13T02:42:39.023 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574913' 2026-02-13T02:42:39.023 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574913 2026-02-13T02:42:39.024 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:42:39.180 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574913 -lt 55834574913 2026-02-13T02:42:39.181 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:42:39.181 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:42:39.181 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:42:39.332 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:42:39.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:42:39.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:42:39.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:42:39.342 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:42:39.342 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:42:39.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:42:39.502 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:42:39.502 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:42:39.502 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:42:39.502 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:39.502 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:42:39.574 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836553 2026-02-13T02:42:39.574 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836553 2026-02-13T02:42:39.574 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836553' 2026-02-13T02:42:39.574 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:39.574 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:42:39.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705734 2026-02-13T02:42:39.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705734 2026-02-13T02:42:39.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836553 1-38654705734' 2026-02-13T02:42:39.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:39.647 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:42:39.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574916 2026-02-13T02:42:39.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574916 2026-02-13T02:42:39.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836553 1-38654705734 2-55834574916' 2026-02-13T02:42:39.718 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:39.718 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836553 2026-02-13T02:42:39.718 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:39.719 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:42:39.720 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836553 2026-02-13T02:42:39.721 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:39.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836553 2026-02-13T02:42:39.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836553' 2026-02-13T02:42:39.722 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836553 2026-02-13T02:42:39.722 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:42:39.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836553 -lt 21474836553 2026-02-13T02:42:39.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:39.894 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705734 2026-02-13T02:42:39.895 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:39.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:42:39.897 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705734 2026-02-13T02:42:39.897 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:39.898 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705734 2026-02-13T02:42:39.898 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705734 2026-02-13T02:42:39.899 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705734' 2026-02-13T02:42:39.899 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:42:40.058 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705734 -lt 38654705734 2026-02-13T02:42:40.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:40.059 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574916 2026-02-13T02:42:40.059 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:40.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:42:40.061 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574916 2026-02-13T02:42:40.062 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:40.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574916 2026-02-13T02:42:40.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574916' 2026-02-13T02:42:40.063 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574916 2026-02-13T02:42:40.063 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:42:40.218 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574915 -lt 55834574916 2026-02-13T02:42:40.219 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:42:41.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:42:41.220 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:42:41.383 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574915 -lt 55834574916 2026-02-13T02:42:41.383 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:42:42.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 499 -eq 0 ']' 2026-02-13T02:42:42.385 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:42:42.548 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574918 -lt 55834574916 2026-02-13T02:42:42.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:42:42.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:42:42.549 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:42:42.696 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:42:42.707 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:42:42.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:42:42.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:42:42.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:42:42.708 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:42:42.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:42:42.876 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:42:42.876 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:42:42.876 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:42:42.876 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:42.876 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:42:42.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836558 2026-02-13T02:42:42.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836558 2026-02-13T02:42:42.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836558' 2026-02-13T02:42:42.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:42.949 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:42:43.021 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705739 2026-02-13T02:42:43.021 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705739 2026-02-13T02:42:43.021 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836558 1-38654705739' 2026-02-13T02:42:43.021 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:43.022 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:42:43.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574920 2026-02-13T02:42:43.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574920 2026-02-13T02:42:43.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836558 1-38654705739 2-55834574920' 2026-02-13T02:42:43.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:43.094 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836558 2026-02-13T02:42:43.094 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:43.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:42:43.096 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836558 2026-02-13T02:42:43.096 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:43.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836558 2026-02-13T02:42:43.097 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836558 2026-02-13T02:42:43.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836558' 2026-02-13T02:42:43.098 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:42:43.266 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836555 -lt 21474836558 2026-02-13T02:42:43.267 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:42:44.267 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:42:44.268 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:42:44.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836558 -lt 21474836558 2026-02-13T02:42:44.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:44.429 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705739 2026-02-13T02:42:44.429 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:44.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:42:44.431 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705739 2026-02-13T02:42:44.431 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:44.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705739 2026-02-13T02:42:44.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705739' 2026-02-13T02:42:44.433 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705739 2026-02-13T02:42:44.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:42:44.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705739 -lt 38654705739 2026-02-13T02:42:44.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:44.606 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574920 2026-02-13T02:42:44.606 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:44.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:42:44.608 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574920 2026-02-13T02:42:44.608 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:44.610 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574920 2026-02-13T02:42:44.610 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574920' 2026-02-13T02:42:44.610 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574920 2026-02-13T02:42:44.610 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:42:44.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574921 -lt 55834574920 2026-02-13T02:42:44.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:42:44.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:42:44.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:42:44.932 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:42:44.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:42:44.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:42:44.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:42:44.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:42:44.942 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:42:45.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:42:45.097 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:42:45.097 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:42:45.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:42:45.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:45.097 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:42:45.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836561 2026-02-13T02:42:45.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836561 2026-02-13T02:42:45.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836561' 2026-02-13T02:42:45.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:45.172 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:42:45.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705742 2026-02-13T02:42:45.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705742 2026-02-13T02:42:45.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836561 1-38654705742' 2026-02-13T02:42:45.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:45.245 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:42:45.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574923 2026-02-13T02:42:45.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574923 2026-02-13T02:42:45.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836561 1-38654705742 2-55834574923' 2026-02-13T02:42:45.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:45.318 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836561 2026-02-13T02:42:45.318 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:45.319 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:42:45.319 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836561 2026-02-13T02:42:45.319 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:45.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836561 2026-02-13T02:42:45.321 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836561 2026-02-13T02:42:45.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836561' 2026-02-13T02:42:45.322 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:42:45.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836558 -lt 21474836561 2026-02-13T02:42:45.493 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:42:46.494 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:42:46.495 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:42:46.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836561 -lt 21474836561 2026-02-13T02:42:46.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:46.655 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705742 2026-02-13T02:42:46.656 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:46.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:42:46.658 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705742 2026-02-13T02:42:46.658 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:46.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705742 2026-02-13T02:42:46.659 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705742' 2026-02-13T02:42:46.659 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705742 2026-02-13T02:42:46.660 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:42:46.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705742 -lt 38654705742 2026-02-13T02:42:46.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:46.821 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574923 2026-02-13T02:42:46.821 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:46.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:42:46.823 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574923 2026-02-13T02:42:46.823 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:46.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574923 2026-02-13T02:42:46.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574923' 2026-02-13T02:42:46.825 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574923 2026-02-13T02:42:46.825 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:42:46.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574924 -lt 55834574923 2026-02-13T02:42:46.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:42:46.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:42:46.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:42:47.138 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:42:47.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:42:47.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:42:47.148 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:42:47.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:42:47.149 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:42:47.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:42:47.309 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:42:47.309 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:42:47.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:42:47.310 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:47.310 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:42:47.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836564 2026-02-13T02:42:47.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836564 2026-02-13T02:42:47.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836564' 2026-02-13T02:42:47.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:47.381 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:42:47.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705745 2026-02-13T02:42:47.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705745 2026-02-13T02:42:47.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836564 1-38654705745' 2026-02-13T02:42:47.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:47.454 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:42:47.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574927 2026-02-13T02:42:47.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574927 2026-02-13T02:42:47.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836564 1-38654705745 2-55834574927' 2026-02-13T02:42:47.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:47.524 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836564 2026-02-13T02:42:47.525 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:47.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:42:47.526 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836564 2026-02-13T02:42:47.527 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:47.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836564 2026-02-13T02:42:47.528 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836564 2026-02-13T02:42:47.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836564' 2026-02-13T02:42:47.529 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:42:47.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836561 -lt 21474836564 2026-02-13T02:42:47.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:42:48.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:42:48.703 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:42:48.860 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836564 -lt 21474836564 2026-02-13T02:42:48.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:48.861 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705745 2026-02-13T02:42:48.861 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:48.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:42:48.863 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705745 2026-02-13T02:42:48.863 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:48.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705745 2026-02-13T02:42:48.865 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705745 2026-02-13T02:42:48.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705745' 2026-02-13T02:42:48.866 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:42:49.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705745 -lt 38654705745 2026-02-13T02:42:49.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:49.037 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574927 2026-02-13T02:42:49.037 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:49.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:42:49.039 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574927 2026-02-13T02:42:49.039 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:49.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574927 2026-02-13T02:42:49.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574927' 2026-02-13T02:42:49.041 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574927 2026-02-13T02:42:49.041 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:42:49.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574927 -lt 55834574927 2026-02-13T02:42:49.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:42:49.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:42:49.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:42:49.362 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:42:49.372 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:42:49.372 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:42:49.372 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:42:49.372 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:42:49.372 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:42:49.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:42:49.530 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:42:49.530 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:42:49.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:42:49.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:49.530 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:42:49.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836567 2026-02-13T02:42:49.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836567 2026-02-13T02:42:49.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836567' 2026-02-13T02:42:49.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:49.603 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:42:49.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705748 2026-02-13T02:42:49.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705748 2026-02-13T02:42:49.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836567 1-38654705748' 2026-02-13T02:42:49.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:49.676 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:42:49.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574930 2026-02-13T02:42:49.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574930 2026-02-13T02:42:49.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836567 1-38654705748 2-55834574930' 2026-02-13T02:42:49.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:49.750 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836567 2026-02-13T02:42:49.750 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:49.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:42:49.751 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836567 2026-02-13T02:42:49.752 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:49.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836567 2026-02-13T02:42:49.753 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836567 2026-02-13T02:42:49.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836567' 2026-02-13T02:42:49.754 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:42:49.924 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836567 -lt 21474836567 2026-02-13T02:42:49.925 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:49.925 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705748 2026-02-13T02:42:49.925 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:49.926 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:42:49.927 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705748 2026-02-13T02:42:49.927 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:49.929 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705748 2026-02-13T02:42:49.929 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705748 2026-02-13T02:42:49.929 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705748' 2026-02-13T02:42:49.930 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:42:50.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705748 -lt 38654705748 2026-02-13T02:42:50.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:50.098 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574930 2026-02-13T02:42:50.098 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:50.099 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:42:50.099 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574930 2026-02-13T02:42:50.100 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:50.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574930 2026-02-13T02:42:50.101 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574930 2026-02-13T02:42:50.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574930' 2026-02-13T02:42:50.102 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:42:50.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574929 -lt 55834574930 2026-02-13T02:42:50.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:42:51.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:42:51.275 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:42:51.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574929 -lt 55834574930 2026-02-13T02:42:51.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:42:52.441 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 499 -eq 0 ']' 2026-02-13T02:42:52.441 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:42:52.597 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574932 -lt 55834574930 2026-02-13T02:42:52.598 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:42:52.598 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:42:52.598 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:42:52.755 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:42:52.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:42:52.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:42:52.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:42:52.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:42:52.766 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:42:52.928 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:42:52.928 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:42:52.928 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:42:52.928 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:42:52.928 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:52.928 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:42:53.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836572 2026-02-13T02:42:53.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836572 2026-02-13T02:42:53.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836572' 2026-02-13T02:42:53.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:53.000 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:42:53.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705753 2026-02-13T02:42:53.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705753 2026-02-13T02:42:53.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836572 1-38654705753' 2026-02-13T02:42:53.075 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:53.075 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:42:53.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574934 2026-02-13T02:42:53.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574934 2026-02-13T02:42:53.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836572 1-38654705753 2-55834574934' 2026-02-13T02:42:53.150 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:53.150 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836572 2026-02-13T02:42:53.151 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:53.152 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:42:53.153 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836572 2026-02-13T02:42:53.153 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:53.154 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836572 2026-02-13T02:42:53.154 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836572' 2026-02-13T02:42:53.154 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836572 2026-02-13T02:42:53.155 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:42:53.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836569 -lt 21474836572 2026-02-13T02:42:53.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:42:54.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:42:54.319 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:42:54.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836572 -lt 21474836572 2026-02-13T02:42:54.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:54.475 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705753 2026-02-13T02:42:54.476 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:54.477 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:42:54.478 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705753 2026-02-13T02:42:54.478 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:54.479 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705753 2026-02-13T02:42:54.480 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705753 2026-02-13T02:42:54.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705753' 2026-02-13T02:42:54.480 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:42:54.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705753 -lt 38654705753 2026-02-13T02:42:54.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:54.653 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574934 2026-02-13T02:42:54.654 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:54.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:42:54.655 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574934 2026-02-13T02:42:54.655 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:54.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574934 2026-02-13T02:42:54.657 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574934 2026-02-13T02:42:54.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574934' 2026-02-13T02:42:54.658 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:42:54.826 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574935 -lt 55834574934 2026-02-13T02:42:54.827 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:42:54.827 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:42:54.827 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:42:54.974 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:42:54.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:42:54.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:42:54.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:42:54.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:42:54.984 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:42:55.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:42:55.146 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:42:55.146 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:42:55.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:42:55.146 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:55.146 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:42:55.218 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836575 2026-02-13T02:42:55.218 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836575 2026-02-13T02:42:55.218 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836575' 2026-02-13T02:42:55.218 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:55.218 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:42:55.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705756 2026-02-13T02:42:55.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705756 2026-02-13T02:42:55.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836575 1-38654705756' 2026-02-13T02:42:55.288 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:55.288 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:42:55.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574937 2026-02-13T02:42:55.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574937 2026-02-13T02:42:55.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836575 1-38654705756 2-55834574937' 2026-02-13T02:42:55.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:55.361 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836575 2026-02-13T02:42:55.361 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:55.362 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:42:55.363 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836575 2026-02-13T02:42:55.363 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:55.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836575 2026-02-13T02:42:55.364 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836575' 2026-02-13T02:42:55.364 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836575 2026-02-13T02:42:55.365 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:42:55.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836572 -lt 21474836575 2026-02-13T02:42:55.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:42:56.535 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:42:56.536 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:42:56.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836575 -lt 21474836575 2026-02-13T02:42:56.701 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:56.701 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705756 2026-02-13T02:42:56.701 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:56.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:42:56.703 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705756 2026-02-13T02:42:56.703 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:56.705 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705756 2026-02-13T02:42:56.705 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705756 2026-02-13T02:42:56.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705756' 2026-02-13T02:42:56.706 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:42:56.876 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705756 -lt 38654705756 2026-02-13T02:42:56.877 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:56.877 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574937 2026-02-13T02:42:56.877 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:56.878 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:42:56.879 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574937 2026-02-13T02:42:56.879 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:56.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574937 2026-02-13T02:42:56.881 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574937 2026-02-13T02:42:56.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574937' 2026-02-13T02:42:56.882 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:42:57.051 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574938 -lt 55834574937 2026-02-13T02:42:57.052 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:42:57.052 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:42:57.052 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:42:57.203 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:42:57.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:42:57.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:42:57.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:42:57.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:42:57.214 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:42:57.376 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:42:57.376 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:42:57.377 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:42:57.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:42:57.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:57.377 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:42:57.447 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836578 2026-02-13T02:42:57.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836578 2026-02-13T02:42:57.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836578' 2026-02-13T02:42:57.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:57.448 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:42:57.520 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705759 2026-02-13T02:42:57.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705759 2026-02-13T02:42:57.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836578 1-38654705759' 2026-02-13T02:42:57.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:57.521 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:42:57.591 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574941 2026-02-13T02:42:57.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574941 2026-02-13T02:42:57.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836578 1-38654705759 2-55834574941' 2026-02-13T02:42:57.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:57.592 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836578 2026-02-13T02:42:57.592 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:57.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:42:57.594 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836578 2026-02-13T02:42:57.595 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:57.596 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836578 2026-02-13T02:42:57.596 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836578' 2026-02-13T02:42:57.596 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836578 2026-02-13T02:42:57.597 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:42:57.764 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836578 -lt 21474836578 2026-02-13T02:42:57.764 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:57.765 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705759 2026-02-13T02:42:57.765 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:57.766 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:42:57.767 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705759 2026-02-13T02:42:57.767 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:57.769 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705759 2026-02-13T02:42:57.769 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705759' 2026-02-13T02:42:57.769 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705759 2026-02-13T02:42:57.770 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:42:57.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705759 -lt 38654705759 2026-02-13T02:42:57.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:57.939 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574941 2026-02-13T02:42:57.940 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:57.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:42:57.941 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574941 2026-02-13T02:42:57.941 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:57.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574941 2026-02-13T02:42:57.943 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574941 2026-02-13T02:42:57.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574941' 2026-02-13T02:42:57.943 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:42:58.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574941 -lt 55834574941 2026-02-13T02:42:58.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:42:58.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:42:58.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:42:58.245 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:42:58.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:42:58.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:42:58.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:42:58.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:42:58.257 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:42:58.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:42:58.421 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:42:58.421 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:42:58.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:42:58.421 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:58.421 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:42:58.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836580 2026-02-13T02:42:58.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836580 2026-02-13T02:42:58.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836580' 2026-02-13T02:42:58.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:58.493 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:42:58.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705761 2026-02-13T02:42:58.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705761 2026-02-13T02:42:58.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836580 1-38654705761' 2026-02-13T02:42:58.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:42:58.565 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:42:58.636 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574943 2026-02-13T02:42:58.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574943 2026-02-13T02:42:58.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836580 1-38654705761 2-55834574943' 2026-02-13T02:42:58.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:58.637 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836580 2026-02-13T02:42:58.638 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:58.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:42:58.639 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836580 2026-02-13T02:42:58.639 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:58.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836580 2026-02-13T02:42:58.641 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836580 2026-02-13T02:42:58.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836580' 2026-02-13T02:42:58.641 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:42:58.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836578 -lt 21474836580 2026-02-13T02:42:58.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:42:59.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:42:59.812 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:42:59.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836581 -lt 21474836580 2026-02-13T02:42:59.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:42:59.970 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705761 2026-02-13T02:42:59.970 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:42:59.971 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:42:59.972 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705761 2026-02-13T02:42:59.972 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:42:59.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705761 2026-02-13T02:42:59.973 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705761 2026-02-13T02:42:59.974 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705761' 2026-02-13T02:42:59.974 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:43:00.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705762 -lt 38654705761 2026-02-13T02:43:00.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:00.150 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574943 2026-02-13T02:43:00.150 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:00.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:43:00.152 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574943 2026-02-13T02:43:00.152 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:00.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574943 2026-02-13T02:43:00.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574943' 2026-02-13T02:43:00.153 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574943 2026-02-13T02:43:00.154 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:43:00.316 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574944 -lt 55834574943 2026-02-13T02:43:00.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:43:00.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:43:00.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:43:00.468 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:43:00.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:43:00.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:43:00.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:43:00.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:43:00.479 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:43:00.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:43:00.641 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:43:00.641 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:43:00.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:43:00.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:00.642 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:43:00.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836583 2026-02-13T02:43:00.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836583 2026-02-13T02:43:00.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836583' 2026-02-13T02:43:00.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:00.717 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:43:00.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705765 2026-02-13T02:43:00.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705765 2026-02-13T02:43:00.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836583 1-38654705765' 2026-02-13T02:43:00.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:00.791 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:43:00.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574946 2026-02-13T02:43:00.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574946 2026-02-13T02:43:00.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836583 1-38654705765 2-55834574946' 2026-02-13T02:43:00.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:00.866 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836583 2026-02-13T02:43:00.866 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:00.867 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:43:00.868 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836583 2026-02-13T02:43:00.868 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:00.869 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836583 2026-02-13T02:43:00.869 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836583' 2026-02-13T02:43:00.869 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836583 2026-02-13T02:43:00.870 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:01.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836581 -lt 21474836583 2026-02-13T02:43:01.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:43:02.042 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:43:02.042 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:02.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836584 -lt 21474836583 2026-02-13T02:43:02.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:02.198 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705765 2026-02-13T02:43:02.198 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:02.199 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:43:02.200 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705765 2026-02-13T02:43:02.200 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:02.202 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705765 2026-02-13T02:43:02.202 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705765' 2026-02-13T02:43:02.202 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705765 2026-02-13T02:43:02.202 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:43:02.376 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705765 -lt 38654705765 2026-02-13T02:43:02.376 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:02.377 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574946 2026-02-13T02:43:02.377 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:02.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:43:02.379 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574946 2026-02-13T02:43:02.379 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:02.381 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574946 2026-02-13T02:43:02.381 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574946 2026-02-13T02:43:02.381 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574946' 2026-02-13T02:43:02.381 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:43:02.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574947 -lt 55834574946 2026-02-13T02:43:02.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:43:02.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:43:02.556 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:43:02.703 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:43:02.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:43:02.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:43:02.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:43:02.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:43:02.712 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:43:02.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:43:02.873 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:43:02.873 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:43:02.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:43:02.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:02.873 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:43:02.946 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836587 2026-02-13T02:43:02.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836587 2026-02-13T02:43:02.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836587' 2026-02-13T02:43:02.947 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:02.947 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:43:03.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705768 2026-02-13T02:43:03.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705768 2026-02-13T02:43:03.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836587 1-38654705768' 2026-02-13T02:43:03.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:03.020 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:43:03.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574949 2026-02-13T02:43:03.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574949 2026-02-13T02:43:03.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836587 1-38654705768 2-55834574949' 2026-02-13T02:43:03.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:03.091 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836587 2026-02-13T02:43:03.091 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:03.093 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:43:03.093 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836587 2026-02-13T02:43:03.094 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:03.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836587 2026-02-13T02:43:03.095 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836587 2026-02-13T02:43:03.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836587' 2026-02-13T02:43:03.095 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:03.270 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836584 -lt 21474836587 2026-02-13T02:43:03.270 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:43:04.271 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:43:04.272 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:04.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836587 -lt 21474836587 2026-02-13T02:43:04.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:04.431 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705768 2026-02-13T02:43:04.432 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:04.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:43:04.434 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705768 2026-02-13T02:43:04.434 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:04.435 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705768 2026-02-13T02:43:04.435 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705768 2026-02-13T02:43:04.435 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705768' 2026-02-13T02:43:04.436 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:43:04.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705768 -lt 38654705768 2026-02-13T02:43:04.607 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:04.608 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574949 2026-02-13T02:43:04.608 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:04.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:43:04.610 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574949 2026-02-13T02:43:04.610 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:04.611 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574949 2026-02-13T02:43:04.611 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574949 2026-02-13T02:43:04.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574949' 2026-02-13T02:43:04.612 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:43:04.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574950 -lt 55834574949 2026-02-13T02:43:04.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:43:04.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:43:04.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:43:04.925 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:43:04.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:43:04.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:43:04.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:43:04.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:43:04.935 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:43:05.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:43:05.095 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:43:05.095 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:43:05.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:43:05.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:05.096 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:43:05.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836590 2026-02-13T02:43:05.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836590 2026-02-13T02:43:05.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836590' 2026-02-13T02:43:05.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:05.167 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:43:05.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705771 2026-02-13T02:43:05.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705771 2026-02-13T02:43:05.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836590 1-38654705771' 2026-02-13T02:43:05.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:05.250 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:43:05.319 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574952 2026-02-13T02:43:05.319 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574952 2026-02-13T02:43:05.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836590 1-38654705771 2-55834574952' 2026-02-13T02:43:05.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:05.320 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836590 2026-02-13T02:43:05.320 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:05.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:43:05.322 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836590 2026-02-13T02:43:05.322 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:05.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836590 2026-02-13T02:43:05.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836590' 2026-02-13T02:43:05.324 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836590 2026-02-13T02:43:05.325 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:05.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836587 -lt 21474836590 2026-02-13T02:43:05.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:43:06.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:43:06.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:06.645 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836590 -lt 21474836590 2026-02-13T02:43:06.645 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:06.645 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705771 2026-02-13T02:43:06.646 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:06.647 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:43:06.648 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705771 2026-02-13T02:43:06.648 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:06.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705771 2026-02-13T02:43:06.649 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705771 2026-02-13T02:43:06.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705771' 2026-02-13T02:43:06.650 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:43:06.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705771 -lt 38654705771 2026-02-13T02:43:06.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:06.819 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574952 2026-02-13T02:43:06.819 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:06.820 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:43:06.821 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574952 2026-02-13T02:43:06.821 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:06.823 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574952 2026-02-13T02:43:06.823 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574952' 2026-02-13T02:43:06.823 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574952 2026-02-13T02:43:06.823 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:43:06.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574953 -lt 55834574952 2026-02-13T02:43:06.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:43:06.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:43:06.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:43:07.127 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:43:07.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:43:07.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:43:07.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:43:07.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:43:07.137 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:43:07.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:43:07.296 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:43:07.296 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:43:07.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:43:07.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:07.296 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:43:07.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836593 2026-02-13T02:43:07.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836593 2026-02-13T02:43:07.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836593' 2026-02-13T02:43:07.371 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:07.371 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:43:07.442 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705774 2026-02-13T02:43:07.442 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705774 2026-02-13T02:43:07.442 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836593 1-38654705774' 2026-02-13T02:43:07.442 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:07.443 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:43:07.513 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574956 2026-02-13T02:43:07.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574956 2026-02-13T02:43:07.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836593 1-38654705774 2-55834574956' 2026-02-13T02:43:07.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:07.514 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836593 2026-02-13T02:43:07.515 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:07.515 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:43:07.516 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836593 2026-02-13T02:43:07.516 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:07.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836593 2026-02-13T02:43:07.517 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836593' 2026-02-13T02:43:07.518 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836593 2026-02-13T02:43:07.518 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:07.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836590 -lt 21474836593 2026-02-13T02:43:07.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:43:08.683 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:43:08.684 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:08.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836593 -lt 21474836593 2026-02-13T02:43:08.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:08.843 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705774 2026-02-13T02:43:08.843 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:08.845 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:43:08.845 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705774 2026-02-13T02:43:08.845 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:08.847 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705774 2026-02-13T02:43:08.847 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705774' 2026-02-13T02:43:08.847 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705774 2026-02-13T02:43:08.847 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:43:09.021 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705774 -lt 38654705774 2026-02-13T02:43:09.021 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:09.022 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574956 2026-02-13T02:43:09.022 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:09.023 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:43:09.024 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574956 2026-02-13T02:43:09.024 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:09.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574956 2026-02-13T02:43:09.026 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574956' 2026-02-13T02:43:09.026 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574956 2026-02-13T02:43:09.026 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:43:09.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574956 -lt 55834574956 2026-02-13T02:43:09.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:43:09.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:43:09.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:43:09.343 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:43:09.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:43:09.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:43:09.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:43:09.353 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:43:09.354 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:43:09.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:43:09.514 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:43:09.514 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:43:09.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:43:09.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:09.515 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:43:09.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836596 2026-02-13T02:43:09.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836596 2026-02-13T02:43:09.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836596' 2026-02-13T02:43:09.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:09.588 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:43:09.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705777 2026-02-13T02:43:09.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705777 2026-02-13T02:43:09.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836596 1-38654705777' 2026-02-13T02:43:09.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:09.661 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:43:09.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574959 2026-02-13T02:43:09.735 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574959 2026-02-13T02:43:09.735 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836596 1-38654705777 2-55834574959' 2026-02-13T02:43:09.735 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:09.735 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836596 2026-02-13T02:43:09.735 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:09.737 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:43:09.737 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836596 2026-02-13T02:43:09.738 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:09.739 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836596 2026-02-13T02:43:09.739 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836596' 2026-02-13T02:43:09.739 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836596 2026-02-13T02:43:09.740 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:09.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836596 -lt 21474836596 2026-02-13T02:43:09.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:09.895 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705777 2026-02-13T02:43:09.895 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:09.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:43:09.897 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705777 2026-02-13T02:43:09.897 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:09.898 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705777 2026-02-13T02:43:09.898 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705777 2026-02-13T02:43:09.899 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705777' 2026-02-13T02:43:09.899 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:43:10.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705777 -lt 38654705777 2026-02-13T02:43:10.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:10.061 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574959 2026-02-13T02:43:10.061 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:10.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:43:10.063 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574959 2026-02-13T02:43:10.063 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:10.065 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574959 2026-02-13T02:43:10.065 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574959 2026-02-13T02:43:10.065 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574959' 2026-02-13T02:43:10.065 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:43:10.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574958 -lt 55834574959 2026-02-13T02:43:10.232 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:43:11.233 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:43:11.233 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:43:11.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574958 -lt 55834574959 2026-02-13T02:43:11.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:43:12.401 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 499 -eq 0 ']' 2026-02-13T02:43:12.402 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:43:12.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574961 -lt 55834574959 2026-02-13T02:43:12.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:43:12.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:43:12.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:43:12.712 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:43:12.723 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:43:12.723 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:43:12.723 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:43:12.723 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:43:12.724 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:43:12.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:43:12.884 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:43:12.884 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:43:12.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:43:12.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:12.884 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:43:12.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836601 2026-02-13T02:43:12.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836601 2026-02-13T02:43:12.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836601' 2026-02-13T02:43:12.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:12.957 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:43:13.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705782 2026-02-13T02:43:13.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705782 2026-02-13T02:43:13.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836601 1-38654705782' 2026-02-13T02:43:13.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:13.031 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:43:13.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574963 2026-02-13T02:43:13.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574963 2026-02-13T02:43:13.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836601 1-38654705782 2-55834574963' 2026-02-13T02:43:13.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:13.107 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836601 2026-02-13T02:43:13.107 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:13.108 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:43:13.109 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836601 2026-02-13T02:43:13.109 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:13.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836601 2026-02-13T02:43:13.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836601' 2026-02-13T02:43:13.111 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836601 2026-02-13T02:43:13.111 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:13.269 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836598 -lt 21474836601 2026-02-13T02:43:13.269 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:43:14.271 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:43:14.271 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:14.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836601 -lt 21474836601 2026-02-13T02:43:14.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:14.437 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705782 2026-02-13T02:43:14.437 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:14.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:43:14.439 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705782 2026-02-13T02:43:14.439 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:14.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705782 2026-02-13T02:43:14.440 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705782' 2026-02-13T02:43:14.441 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705782 2026-02-13T02:43:14.441 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:43:14.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705782 -lt 38654705782 2026-02-13T02:43:14.609 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:14.610 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574963 2026-02-13T02:43:14.610 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:14.611 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:43:14.612 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574963 2026-02-13T02:43:14.612 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:14.613 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574963 2026-02-13T02:43:14.613 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574963' 2026-02-13T02:43:14.614 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574963 2026-02-13T02:43:14.614 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:43:14.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574964 -lt 55834574963 2026-02-13T02:43:14.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:43:14.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:43:14.786 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:43:14.931 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:43:14.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:43:14.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:43:14.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:43:14.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:43:14.943 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:43:15.099 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:43:15.099 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:43:15.100 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:43:15.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:43:15.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:15.100 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:43:15.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836604 2026-02-13T02:43:15.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836604 2026-02-13T02:43:15.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836604' 2026-02-13T02:43:15.172 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:15.172 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:43:15.243 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705785 2026-02-13T02:43:15.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705785 2026-02-13T02:43:15.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836604 1-38654705785' 2026-02-13T02:43:15.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:15.244 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:43:15.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574966 2026-02-13T02:43:15.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574966 2026-02-13T02:43:15.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836604 1-38654705785 2-55834574966' 2026-02-13T02:43:15.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:15.321 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836604 2026-02-13T02:43:15.321 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:15.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:43:15.322 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836604 2026-02-13T02:43:15.322 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:15.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836604 2026-02-13T02:43:15.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836604' 2026-02-13T02:43:15.324 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836604 2026-02-13T02:43:15.325 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:15.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836601 -lt 21474836604 2026-02-13T02:43:15.490 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:43:16.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:43:16.492 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:16.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836604 -lt 21474836604 2026-02-13T02:43:16.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:16.653 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705785 2026-02-13T02:43:16.653 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:16.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:43:16.655 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705785 2026-02-13T02:43:16.655 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:16.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705785 2026-02-13T02:43:16.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705785' 2026-02-13T02:43:16.657 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705785 2026-02-13T02:43:16.657 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:43:16.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705785 -lt 38654705785 2026-02-13T02:43:16.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:16.830 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574966 2026-02-13T02:43:16.831 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:16.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:43:16.833 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574966 2026-02-13T02:43:16.833 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:16.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574966 2026-02-13T02:43:16.834 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574966 2026-02-13T02:43:16.835 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574966' 2026-02-13T02:43:16.835 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:43:16.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574967 -lt 55834574966 2026-02-13T02:43:16.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:43:16.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:43:16.998 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:43:17.143 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:43:17.152 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:43:17.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:43:17.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:43:17.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:43:17.153 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:43:17.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:43:17.308 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:43:17.308 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:43:17.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:43:17.309 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:17.309 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:43:17.381 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836607 2026-02-13T02:43:17.381 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836607 2026-02-13T02:43:17.382 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836607' 2026-02-13T02:43:17.382 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:17.382 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:43:17.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705788 2026-02-13T02:43:17.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705788 2026-02-13T02:43:17.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836607 1-38654705788' 2026-02-13T02:43:17.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:17.453 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:43:17.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574970 2026-02-13T02:43:17.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574970 2026-02-13T02:43:17.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836607 1-38654705788 2-55834574970' 2026-02-13T02:43:17.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:17.522 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836607 2026-02-13T02:43:17.522 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:17.523 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:43:17.524 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836607 2026-02-13T02:43:17.524 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:17.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836607 2026-02-13T02:43:17.526 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836607 2026-02-13T02:43:17.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836607' 2026-02-13T02:43:17.526 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:17.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836604 -lt 21474836607 2026-02-13T02:43:17.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:43:18.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:43:18.692 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:18.850 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836607 -lt 21474836607 2026-02-13T02:43:18.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:18.851 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705788 2026-02-13T02:43:18.851 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:18.852 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:43:18.853 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705788 2026-02-13T02:43:18.853 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:18.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705788 2026-02-13T02:43:18.855 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705788 2026-02-13T02:43:18.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705788' 2026-02-13T02:43:18.855 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:43:19.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705788 -lt 38654705788 2026-02-13T02:43:19.015 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:19.015 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574970 2026-02-13T02:43:19.015 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:19.017 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:43:19.017 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574970 2026-02-13T02:43:19.018 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:19.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574970 2026-02-13T02:43:19.019 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574970 2026-02-13T02:43:19.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574970' 2026-02-13T02:43:19.020 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:43:19.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574970 -lt 55834574970 2026-02-13T02:43:19.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:43:19.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:43:19.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:43:19.335 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:43:19.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:43:19.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:43:19.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:43:19.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:43:19.345 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:43:19.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:43:19.500 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:43:19.500 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:43:19.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:43:19.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:19.501 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:43:19.575 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836610 2026-02-13T02:43:19.575 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836610 2026-02-13T02:43:19.575 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836610' 2026-02-13T02:43:19.575 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:19.575 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:43:19.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705791 2026-02-13T02:43:19.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705791 2026-02-13T02:43:19.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836610 1-38654705791' 2026-02-13T02:43:19.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:19.652 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:43:19.723 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574973 2026-02-13T02:43:19.723 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574973 2026-02-13T02:43:19.723 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836610 1-38654705791 2-55834574973' 2026-02-13T02:43:19.723 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:19.724 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836610 2026-02-13T02:43:19.724 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:19.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:43:19.726 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836610 2026-02-13T02:43:19.726 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:19.727 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836610 2026-02-13T02:43:19.727 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836610 2026-02-13T02:43:19.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836610' 2026-02-13T02:43:19.728 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:19.899 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836610 -lt 21474836610 2026-02-13T02:43:19.899 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:19.900 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705791 2026-02-13T02:43:19.900 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:19.901 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:43:19.902 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705791 2026-02-13T02:43:19.902 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:19.903 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705791 2026-02-13T02:43:19.904 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705791' 2026-02-13T02:43:19.904 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705791 2026-02-13T02:43:19.905 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:43:20.072 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705791 -lt 38654705791 2026-02-13T02:43:20.072 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:20.072 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574973 2026-02-13T02:43:20.073 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:20.074 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:43:20.075 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574973 2026-02-13T02:43:20.075 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:20.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574973 2026-02-13T02:43:20.076 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574973' 2026-02-13T02:43:20.076 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574973 2026-02-13T02:43:20.077 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:43:20.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574972 -lt 55834574973 2026-02-13T02:43:20.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:43:21.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:43:21.250 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:43:21.416 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574972 -lt 55834574973 2026-02-13T02:43:21.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:43:22.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 499 -eq 0 ']' 2026-02-13T02:43:22.426 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:43:22.576 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574975 -lt 55834574973 2026-02-13T02:43:22.576 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:43:22.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:43:22.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:43:22.728 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:43:22.737 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:43:22.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:43:22.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:43:22.738 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:43:22.738 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:43:22.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:43:22.897 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:43:22.898 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:43:22.898 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:43:22.898 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:22.898 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:43:22.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836615 2026-02-13T02:43:22.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836615 2026-02-13T02:43:22.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836615' 2026-02-13T02:43:22.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:22.969 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:43:23.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705796 2026-02-13T02:43:23.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705796 2026-02-13T02:43:23.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836615 1-38654705796' 2026-02-13T02:43:23.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:23.041 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:43:23.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574977 2026-02-13T02:43:23.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574977 2026-02-13T02:43:23.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836615 1-38654705796 2-55834574977' 2026-02-13T02:43:23.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:23.112 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836615 2026-02-13T02:43:23.113 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:23.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:43:23.114 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836615 2026-02-13T02:43:23.115 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:23.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836615 2026-02-13T02:43:23.116 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836615 2026-02-13T02:43:23.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836615' 2026-02-13T02:43:23.117 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:23.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836612 -lt 21474836615 2026-02-13T02:43:23.285 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:43:24.286 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:43:24.287 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:24.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836615 -lt 21474836615 2026-02-13T02:43:24.444 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:24.445 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705796 2026-02-13T02:43:24.445 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:24.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:43:24.447 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705796 2026-02-13T02:43:24.447 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:24.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705796 2026-02-13T02:43:24.449 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705796 2026-02-13T02:43:24.449 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705796' 2026-02-13T02:43:24.449 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:43:24.619 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705796 -lt 38654705796 2026-02-13T02:43:24.619 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:24.619 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574977 2026-02-13T02:43:24.620 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:24.621 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:43:24.622 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574977 2026-02-13T02:43:24.622 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:24.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574977 2026-02-13T02:43:24.624 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574977 2026-02-13T02:43:24.624 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574977' 2026-02-13T02:43:24.624 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:43:24.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574978 -lt 55834574977 2026-02-13T02:43:24.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:43:24.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:43:24.794 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:43:24.945 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:43:24.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:43:24.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:43:24.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:43:24.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:43:24.956 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:43:25.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:43:25.116 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:43:25.116 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:43:25.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:43:25.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:25.117 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:43:25.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836618 2026-02-13T02:43:25.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836618 2026-02-13T02:43:25.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836618' 2026-02-13T02:43:25.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:25.188 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:43:25.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705799 2026-02-13T02:43:25.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705799 2026-02-13T02:43:25.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836618 1-38654705799' 2026-02-13T02:43:25.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:25.261 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:43:25.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574980 2026-02-13T02:43:25.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574980 2026-02-13T02:43:25.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836618 1-38654705799 2-55834574980' 2026-02-13T02:43:25.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:25.334 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836618 2026-02-13T02:43:25.334 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:25.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:43:25.335 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836618 2026-02-13T02:43:25.336 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:25.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836618 2026-02-13T02:43:25.337 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836618' 2026-02-13T02:43:25.337 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836618 2026-02-13T02:43:25.338 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:25.510 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836615 -lt 21474836618 2026-02-13T02:43:25.511 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:43:26.512 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:43:26.512 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:26.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836618 -lt 21474836618 2026-02-13T02:43:26.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:26.671 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705799 2026-02-13T02:43:26.671 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:26.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:43:26.673 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705799 2026-02-13T02:43:26.674 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:26.675 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705799 2026-02-13T02:43:26.675 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705799 2026-02-13T02:43:26.676 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705799' 2026-02-13T02:43:26.676 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:43:26.836 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705799 -lt 38654705799 2026-02-13T02:43:26.836 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:26.837 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574980 2026-02-13T02:43:26.837 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:26.838 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:43:26.838 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574980 2026-02-13T02:43:26.839 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:26.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574980 2026-02-13T02:43:26.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574980' 2026-02-13T02:43:26.841 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574980 2026-02-13T02:43:26.841 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:43:27.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574981 -lt 55834574980 2026-02-13T02:43:27.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:43:27.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:43:27.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:43:27.158 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:43:27.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:43:27.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:43:27.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:43:27.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:43:27.168 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:43:27.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:43:27.324 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:43:27.325 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:43:27.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:43:27.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:27.325 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:43:27.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836621 2026-02-13T02:43:27.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836621 2026-02-13T02:43:27.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836621' 2026-02-13T02:43:27.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:27.401 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:43:27.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705802 2026-02-13T02:43:27.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705802 2026-02-13T02:43:27.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836621 1-38654705802' 2026-02-13T02:43:27.473 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:27.473 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:43:27.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574984 2026-02-13T02:43:27.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574984 2026-02-13T02:43:27.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836621 1-38654705802 2-55834574984' 2026-02-13T02:43:27.547 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:27.547 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836621 2026-02-13T02:43:27.547 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:27.548 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:43:27.549 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836621 2026-02-13T02:43:27.549 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:27.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836621 2026-02-13T02:43:27.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836621' 2026-02-13T02:43:27.551 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836621 2026-02-13T02:43:27.551 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:27.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836618 -lt 21474836621 2026-02-13T02:43:27.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:43:28.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:43:28.714 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:28.876 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836621 -lt 21474836621 2026-02-13T02:43:28.876 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:28.877 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705802 2026-02-13T02:43:28.877 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:28.878 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:43:28.879 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705802 2026-02-13T02:43:28.879 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:28.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705802 2026-02-13T02:43:28.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705802' 2026-02-13T02:43:28.881 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705802 2026-02-13T02:43:28.881 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:43:29.057 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705802 -lt 38654705802 2026-02-13T02:43:29.057 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:29.058 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574984 2026-02-13T02:43:29.058 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:29.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:43:29.060 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574984 2026-02-13T02:43:29.060 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:29.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574984 2026-02-13T02:43:29.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574984' 2026-02-13T02:43:29.062 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574984 2026-02-13T02:43:29.062 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:43:29.219 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574984 -lt 55834574984 2026-02-13T02:43:29.219 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:43:29.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:43:29.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:43:29.373 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:43:29.382 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:43:29.382 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:43:29.383 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:43:29.383 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:43:29.383 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:43:29.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:43:29.543 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:43:29.543 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:43:29.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:43:29.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:29.543 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:43:29.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836624 2026-02-13T02:43:29.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836624 2026-02-13T02:43:29.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836624' 2026-02-13T02:43:29.616 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:29.617 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:43:29.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705805 2026-02-13T02:43:29.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705805 2026-02-13T02:43:29.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836624 1-38654705805' 2026-02-13T02:43:29.690 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:29.690 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:43:29.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574987 2026-02-13T02:43:29.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574987 2026-02-13T02:43:29.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836624 1-38654705805 2-55834574987' 2026-02-13T02:43:29.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:29.765 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836624 2026-02-13T02:43:29.766 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:29.767 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:43:29.767 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836624 2026-02-13T02:43:29.768 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:29.769 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836624 2026-02-13T02:43:29.769 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836624 2026-02-13T02:43:29.770 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836624' 2026-02-13T02:43:29.770 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:29.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836624 -lt 21474836624 2026-02-13T02:43:29.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:29.933 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705805 2026-02-13T02:43:29.934 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:29.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:43:29.935 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705805 2026-02-13T02:43:29.936 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:29.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705805 2026-02-13T02:43:29.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705805' 2026-02-13T02:43:29.937 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705805 2026-02-13T02:43:29.938 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:43:30.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705804 -lt 38654705805 2026-02-13T02:43:30.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:43:31.113 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:43:31.113 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:43:31.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705804 -lt 38654705805 2026-02-13T02:43:31.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:43:32.274 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 499 -eq 0 ']' 2026-02-13T02:43:32.275 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:43:32.441 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705807 -lt 38654705805 2026-02-13T02:43:32.441 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:32.441 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574987 2026-02-13T02:43:32.441 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:32.443 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:43:32.444 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574987 2026-02-13T02:43:32.444 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:32.445 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574987 2026-02-13T02:43:32.445 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574987 2026-02-13T02:43:32.446 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574987' 2026-02-13T02:43:32.446 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:43:32.611 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574989 -lt 55834574987 2026-02-13T02:43:32.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:43:32.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:43:32.612 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:43:32.761 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:43:32.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:43:32.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:43:32.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:43:32.772 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:43:32.772 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:43:32.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:43:32.933 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:43:32.934 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:43:32.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:43:32.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:32.934 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:43:33.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836629 2026-02-13T02:43:33.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836629 2026-02-13T02:43:33.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836629' 2026-02-13T02:43:33.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:33.008 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:43:33.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705810 2026-02-13T02:43:33.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705810 2026-02-13T02:43:33.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836629 1-38654705810' 2026-02-13T02:43:33.077 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:33.078 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:43:33.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574991 2026-02-13T02:43:33.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574991 2026-02-13T02:43:33.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836629 1-38654705810 2-55834574991' 2026-02-13T02:43:33.149 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:33.150 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836629 2026-02-13T02:43:33.150 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:33.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:43:33.152 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836629 2026-02-13T02:43:33.152 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:33.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836629 2026-02-13T02:43:33.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836629' 2026-02-13T02:43:33.154 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836629 2026-02-13T02:43:33.154 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:33.319 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836626 -lt 21474836629 2026-02-13T02:43:33.319 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:43:34.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:43:34.321 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:34.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836629 -lt 21474836629 2026-02-13T02:43:34.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:34.484 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705810 2026-02-13T02:43:34.484 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:34.485 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:43:34.486 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705810 2026-02-13T02:43:34.486 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:34.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705810 2026-02-13T02:43:34.487 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705810' 2026-02-13T02:43:34.488 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705810 2026-02-13T02:43:34.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:43:34.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705810 -lt 38654705810 2026-02-13T02:43:34.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:34.660 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574991 2026-02-13T02:43:34.661 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:34.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:43:34.663 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574991 2026-02-13T02:43:34.663 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:34.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574991 2026-02-13T02:43:34.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574991' 2026-02-13T02:43:34.665 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574991 2026-02-13T02:43:34.665 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:43:34.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574992 -lt 55834574991 2026-02-13T02:43:34.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:43:34.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:43:34.831 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:43:34.983 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:43:34.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:43:34.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:43:34.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:43:34.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:43:34.994 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:43:35.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:43:35.153 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:43:35.153 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:43:35.153 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:43:35.154 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:35.154 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:43:35.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836632 2026-02-13T02:43:35.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836632 2026-02-13T02:43:35.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836632' 2026-02-13T02:43:35.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:35.226 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:43:35.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705813 2026-02-13T02:43:35.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705813 2026-02-13T02:43:35.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836632 1-38654705813' 2026-02-13T02:43:35.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:35.300 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:43:35.372 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574994 2026-02-13T02:43:35.372 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574994 2026-02-13T02:43:35.372 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836632 1-38654705813 2-55834574994' 2026-02-13T02:43:35.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:35.373 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836632 2026-02-13T02:43:35.373 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:35.373 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:43:35.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836632 2026-02-13T02:43:35.374 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:35.376 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836632 2026-02-13T02:43:35.376 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836632 2026-02-13T02:43:35.376 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836632' 2026-02-13T02:43:35.377 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:35.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836629 -lt 21474836632 2026-02-13T02:43:35.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:43:36.547 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:43:36.548 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:36.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836632 -lt 21474836632 2026-02-13T02:43:36.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:36.709 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705813 2026-02-13T02:43:36.709 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:36.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:43:36.711 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705813 2026-02-13T02:43:36.711 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:36.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705813 2026-02-13T02:43:36.713 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705813 2026-02-13T02:43:36.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705813' 2026-02-13T02:43:36.713 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:43:36.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705813 -lt 38654705813 2026-02-13T02:43:36.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:36.882 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574994 2026-02-13T02:43:36.882 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:36.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:43:36.884 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574994 2026-02-13T02:43:36.884 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:36.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574994 2026-02-13T02:43:36.886 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574994 2026-02-13T02:43:36.886 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574994' 2026-02-13T02:43:36.886 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:43:37.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574995 -lt 55834574994 2026-02-13T02:43:37.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:43:37.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:43:37.063 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:43:37.213 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:43:37.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:43:37.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:43:37.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:43:37.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:43:37.225 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:43:37.383 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:43:37.383 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:43:37.383 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:43:37.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:43:37.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:37.384 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:43:37.456 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836635 2026-02-13T02:43:37.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836635 2026-02-13T02:43:37.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836635' 2026-02-13T02:43:37.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:37.457 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:43:37.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705816 2026-02-13T02:43:37.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705816 2026-02-13T02:43:37.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836635 1-38654705816' 2026-02-13T02:43:37.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:37.529 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:43:37.601 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574998 2026-02-13T02:43:37.601 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574998 2026-02-13T02:43:37.601 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836635 1-38654705816 2-55834574998' 2026-02-13T02:43:37.601 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:37.602 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836635 2026-02-13T02:43:37.602 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:37.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:43:37.603 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836635 2026-02-13T02:43:37.604 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:37.605 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836635 2026-02-13T02:43:37.605 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836635 2026-02-13T02:43:37.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836635' 2026-02-13T02:43:37.606 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:37.773 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836635 -lt 21474836635 2026-02-13T02:43:37.773 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:37.774 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705816 2026-02-13T02:43:37.774 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:37.775 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:43:37.775 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705816 2026-02-13T02:43:37.775 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:37.777 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705816 2026-02-13T02:43:37.777 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705816 2026-02-13T02:43:37.778 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705816' 2026-02-13T02:43:37.778 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:43:37.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705816 -lt 38654705816 2026-02-13T02:43:37.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:37.949 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574998 2026-02-13T02:43:37.950 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:37.951 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:43:37.951 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574998 2026-02-13T02:43:37.952 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:37.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574998 2026-02-13T02:43:37.953 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834574998 2026-02-13T02:43:37.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574998' 2026-02-13T02:43:37.954 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:43:38.123 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574998 -lt 55834574998 2026-02-13T02:43:38.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:43:38.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:43:38.125 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:43:38.273 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:43:38.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:43:38.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:43:38.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:43:38.283 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:43:38.283 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:43:38.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:43:38.438 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:43:38.438 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:43:38.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:43:38.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:38.438 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:43:38.513 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836637 2026-02-13T02:43:38.513 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836637 2026-02-13T02:43:38.513 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836637' 2026-02-13T02:43:38.513 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:38.514 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:43:38.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705818 2026-02-13T02:43:38.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705818 2026-02-13T02:43:38.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836637 1-38654705818' 2026-02-13T02:43:38.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:38.587 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:43:38.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575000 2026-02-13T02:43:38.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575000 2026-02-13T02:43:38.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836637 1-38654705818 2-55834575000' 2026-02-13T02:43:38.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:38.662 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836637 2026-02-13T02:43:38.662 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:38.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:43:38.664 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836637 2026-02-13T02:43:38.664 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:38.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836637 2026-02-13T02:43:38.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836637' 2026-02-13T02:43:38.666 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836637 2026-02-13T02:43:38.666 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:38.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836635 -lt 21474836637 2026-02-13T02:43:38.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:43:39.835 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:43:39.836 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:39.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836638 -lt 21474836637 2026-02-13T02:43:39.998 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:39.998 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705818 2026-02-13T02:43:39.998 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:39.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:43:40.000 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705818 2026-02-13T02:43:40.000 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:40.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705818 2026-02-13T02:43:40.002 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705818 2026-02-13T02:43:40.002 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705818' 2026-02-13T02:43:40.003 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:43:40.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705819 -lt 38654705818 2026-02-13T02:43:40.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:40.164 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575000 2026-02-13T02:43:40.165 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:40.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:43:40.167 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575000 2026-02-13T02:43:40.167 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:40.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575000 2026-02-13T02:43:40.168 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834575000 2026-02-13T02:43:40.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575000' 2026-02-13T02:43:40.169 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:43:40.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575001 -lt 55834575000 2026-02-13T02:43:40.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:43:40.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:43:40.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:43:40.477 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:43:40.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:43:40.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:43:40.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:43:40.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:43:40.489 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:43:40.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:43:40.655 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:43:40.655 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:43:40.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:43:40.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:40.655 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:43:40.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836640 2026-02-13T02:43:40.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836640 2026-02-13T02:43:40.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836640' 2026-02-13T02:43:40.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:40.733 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:43:40.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705822 2026-02-13T02:43:40.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705822 2026-02-13T02:43:40.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836640 1-38654705822' 2026-02-13T02:43:40.804 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:40.805 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:43:40.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575003 2026-02-13T02:43:40.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575003 2026-02-13T02:43:40.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836640 1-38654705822 2-55834575003' 2026-02-13T02:43:40.875 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:40.875 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836640 2026-02-13T02:43:40.875 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:40.876 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:43:40.877 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836640 2026-02-13T02:43:40.877 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:40.878 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836640 2026-02-13T02:43:40.879 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836640 2026-02-13T02:43:40.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836640' 2026-02-13T02:43:40.879 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:41.048 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836638 -lt 21474836640 2026-02-13T02:43:41.048 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:43:42.050 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:43:42.050 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:42.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836641 -lt 21474836640 2026-02-13T02:43:42.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:42.209 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705822 2026-02-13T02:43:42.209 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:42.210 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:43:42.211 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705822 2026-02-13T02:43:42.211 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:42.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705822 2026-02-13T02:43:42.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705822' 2026-02-13T02:43:42.213 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705822 2026-02-13T02:43:42.214 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:43:42.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705822 -lt 38654705822 2026-02-13T02:43:42.385 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:42.386 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575003 2026-02-13T02:43:42.386 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:42.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:43:42.387 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575003 2026-02-13T02:43:42.387 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:42.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575003 2026-02-13T02:43:42.389 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834575003 2026-02-13T02:43:42.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575003' 2026-02-13T02:43:42.390 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:43:42.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575004 -lt 55834575003 2026-02-13T02:43:42.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:43:42.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:43:42.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:43:42.710 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:43:42.720 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:43:42.720 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:43:42.720 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:43:42.720 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:43:42.720 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:43:42.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:43:42.884 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:43:42.884 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:43:42.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:43:42.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:42.884 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:43:42.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836644 2026-02-13T02:43:42.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836644 2026-02-13T02:43:42.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836644' 2026-02-13T02:43:42.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:42.957 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:43:43.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705825 2026-02-13T02:43:43.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705825 2026-02-13T02:43:43.029 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836644 1-38654705825' 2026-02-13T02:43:43.030 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:43.030 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:43:43.099 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575006 2026-02-13T02:43:43.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575006 2026-02-13T02:43:43.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836644 1-38654705825 2-55834575006' 2026-02-13T02:43:43.100 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:43.100 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836644 2026-02-13T02:43:43.100 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:43.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:43:43.102 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836644 2026-02-13T02:43:43.102 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:43.104 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836644 2026-02-13T02:43:43.104 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836644 2026-02-13T02:43:43.105 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836644' 2026-02-13T02:43:43.105 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:43.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836641 -lt 21474836644 2026-02-13T02:43:43.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:43:44.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:43:44.266 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:44.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836644 -lt 21474836644 2026-02-13T02:43:44.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:44.430 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705825 2026-02-13T02:43:44.430 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:44.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:43:44.432 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705825 2026-02-13T02:43:44.432 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:44.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705825 2026-02-13T02:43:44.434 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705825 2026-02-13T02:43:44.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705825' 2026-02-13T02:43:44.434 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:43:44.598 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705825 -lt 38654705825 2026-02-13T02:43:44.598 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:44.598 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575006 2026-02-13T02:43:44.599 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:44.600 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:43:44.600 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575006 2026-02-13T02:43:44.600 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:44.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575006 2026-02-13T02:43:44.602 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834575006 2026-02-13T02:43:44.602 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575006' 2026-02-13T02:43:44.602 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:43:44.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575007 -lt 55834575006 2026-02-13T02:43:44.766 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:43:44.766 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:43:44.766 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:43:44.913 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:43:44.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:43:44.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:43:44.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:43:44.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:43:44.923 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:43:45.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:43:45.084 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:43:45.084 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:43:45.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:43:45.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:45.085 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:43:45.155 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836647 2026-02-13T02:43:45.155 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836647 2026-02-13T02:43:45.155 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836647' 2026-02-13T02:43:45.155 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:45.156 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:43:45.228 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705828 2026-02-13T02:43:45.228 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705828 2026-02-13T02:43:45.228 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836647 1-38654705828' 2026-02-13T02:43:45.228 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:45.229 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:43:45.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575009 2026-02-13T02:43:45.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575009 2026-02-13T02:43:45.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836647 1-38654705828 2-55834575009' 2026-02-13T02:43:45.302 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:45.302 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836647 2026-02-13T02:43:45.302 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:45.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:43:45.304 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836647 2026-02-13T02:43:45.304 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:45.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836647 2026-02-13T02:43:45.306 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836647 2026-02-13T02:43:45.306 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836647' 2026-02-13T02:43:45.307 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:45.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836644 -lt 21474836647 2026-02-13T02:43:45.470 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:43:46.472 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:43:46.472 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:46.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836647 -lt 21474836647 2026-02-13T02:43:46.630 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:46.630 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705828 2026-02-13T02:43:46.630 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:46.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:43:46.632 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705828 2026-02-13T02:43:46.633 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:46.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705828 2026-02-13T02:43:46.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705828' 2026-02-13T02:43:46.634 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705828 2026-02-13T02:43:46.635 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:43:46.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705828 -lt 38654705828 2026-02-13T02:43:46.799 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:46.799 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575009 2026-02-13T02:43:46.799 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:46.801 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:43:46.801 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575009 2026-02-13T02:43:46.801 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:46.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575009 2026-02-13T02:43:46.803 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575009' 2026-02-13T02:43:46.803 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834575009 2026-02-13T02:43:46.804 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:43:46.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575010 -lt 55834575009 2026-02-13T02:43:46.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:43:46.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:43:46.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:43:47.112 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:43:47.121 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:43:47.122 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:43:47.122 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:43:47.122 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:43:47.122 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:43:47.279 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:43:47.279 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:43:47.279 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:43:47.280 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:43:47.280 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:47.280 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:43:47.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836650 2026-02-13T02:43:47.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836650 2026-02-13T02:43:47.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836650' 2026-02-13T02:43:47.354 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:47.354 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:43:47.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705831 2026-02-13T02:43:47.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705831 2026-02-13T02:43:47.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836650 1-38654705831' 2026-02-13T02:43:47.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:47.425 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:43:47.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575013 2026-02-13T02:43:47.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575013 2026-02-13T02:43:47.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836650 1-38654705831 2-55834575013' 2026-02-13T02:43:47.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:47.500 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836650 2026-02-13T02:43:47.500 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:47.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:43:47.502 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836650 2026-02-13T02:43:47.502 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:47.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836650 2026-02-13T02:43:47.503 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836650 2026-02-13T02:43:47.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836650' 2026-02-13T02:43:47.504 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:47.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836647 -lt 21474836650 2026-02-13T02:43:47.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:43:48.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:43:48.674 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:48.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836650 -lt 21474836650 2026-02-13T02:43:48.840 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:48.841 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705831 2026-02-13T02:43:48.841 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:48.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:43:48.843 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705831 2026-02-13T02:43:48.843 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:48.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705831 2026-02-13T02:43:48.844 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705831 2026-02-13T02:43:48.845 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705831' 2026-02-13T02:43:48.845 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:43:49.013 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705831 -lt 38654705831 2026-02-13T02:43:49.014 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:49.014 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575013 2026-02-13T02:43:49.014 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:49.016 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:43:49.016 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575013 2026-02-13T02:43:49.017 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:49.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575013 2026-02-13T02:43:49.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575013' 2026-02-13T02:43:49.019 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834575013 2026-02-13T02:43:49.019 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:43:49.180 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575013 -lt 55834575013 2026-02-13T02:43:49.180 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:43:49.181 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:43:49.181 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:43:49.329 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:43:49.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:43:49.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:43:49.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:43:49.341 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:43:49.341 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:43:49.505 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:43:49.505 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:43:49.505 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:43:49.505 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:43:49.505 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:49.505 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:43:49.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836653 2026-02-13T02:43:49.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836653 2026-02-13T02:43:49.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836653' 2026-02-13T02:43:49.577 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:49.577 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:43:49.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705834 2026-02-13T02:43:49.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705834 2026-02-13T02:43:49.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836653 1-38654705834' 2026-02-13T02:43:49.649 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:49.649 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:43:49.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575016 2026-02-13T02:43:49.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575016 2026-02-13T02:43:49.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836653 1-38654705834 2-55834575016' 2026-02-13T02:43:49.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:49.722 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836653 2026-02-13T02:43:49.722 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:49.723 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:43:49.724 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836653 2026-02-13T02:43:49.724 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:49.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836653 2026-02-13T02:43:49.725 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836653 2026-02-13T02:43:49.726 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836653' 2026-02-13T02:43:49.726 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:49.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836653 -lt 21474836653 2026-02-13T02:43:49.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:49.886 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705834 2026-02-13T02:43:49.886 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:49.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:43:49.888 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705834 2026-02-13T02:43:49.888 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:49.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705834 2026-02-13T02:43:49.889 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705834 2026-02-13T02:43:49.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705834' 2026-02-13T02:43:49.890 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:43:50.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705834 -lt 38654705834 2026-02-13T02:43:50.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:50.060 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575016 2026-02-13T02:43:50.060 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:50.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:43:50.062 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575016 2026-02-13T02:43:50.062 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:50.064 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575016 2026-02-13T02:43:50.064 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834575016 2026-02-13T02:43:50.064 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575016' 2026-02-13T02:43:50.065 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:43:50.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575015 -lt 55834575016 2026-02-13T02:43:50.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:43:51.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:43:51.238 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:43:51.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575015 -lt 55834575016 2026-02-13T02:43:51.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:43:52.397 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 499 -eq 0 ']' 2026-02-13T02:43:52.397 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:43:52.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575018 -lt 55834575016 2026-02-13T02:43:52.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:43:52.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:43:52.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:43:52.717 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:43:52.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:43:52.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:43:52.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:43:52.728 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:43:52.728 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:43:52.892 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:43:52.893 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:43:52.893 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:43:52.893 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:43:52.893 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:52.893 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:43:52.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836658 2026-02-13T02:43:52.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836658 2026-02-13T02:43:52.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836658' 2026-02-13T02:43:52.964 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:52.965 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:43:53.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705839 2026-02-13T02:43:53.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705839 2026-02-13T02:43:53.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836658 1-38654705839' 2026-02-13T02:43:53.040 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:53.040 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:43:53.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575020 2026-02-13T02:43:53.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575020 2026-02-13T02:43:53.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836658 1-38654705839 2-55834575020' 2026-02-13T02:43:53.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:53.112 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836658 2026-02-13T02:43:53.113 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:53.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:43:53.114 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836658 2026-02-13T02:43:53.115 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:53.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836658 2026-02-13T02:43:53.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836658' 2026-02-13T02:43:53.116 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836658 2026-02-13T02:43:53.117 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:53.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836655 -lt 21474836658 2026-02-13T02:43:53.281 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:43:54.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:43:54.283 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:54.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836658 -lt 21474836658 2026-02-13T02:43:54.448 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:54.449 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705839 2026-02-13T02:43:54.449 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:54.450 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:43:54.451 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705839 2026-02-13T02:43:54.451 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:54.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705839 2026-02-13T02:43:54.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705839' 2026-02-13T02:43:54.453 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705839 2026-02-13T02:43:54.453 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:43:54.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705839 -lt 38654705839 2026-02-13T02:43:54.618 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:54.618 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575020 2026-02-13T02:43:54.618 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:54.619 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:43:54.620 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575020 2026-02-13T02:43:54.620 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:54.621 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575020 2026-02-13T02:43:54.622 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834575020 2026-02-13T02:43:54.622 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575020' 2026-02-13T02:43:54.622 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:43:54.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575021 -lt 55834575020 2026-02-13T02:43:54.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:43:54.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:43:54.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:43:54.938 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:43:54.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:43:54.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:43:54.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:43:54.950 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:43:54.950 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:43:55.114 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:43:55.114 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:43:55.115 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:43:55.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:43:55.115 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:55.115 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:43:55.185 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836661 2026-02-13T02:43:55.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836661 2026-02-13T02:43:55.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836661' 2026-02-13T02:43:55.186 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:55.186 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:43:55.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705842 2026-02-13T02:43:55.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705842 2026-02-13T02:43:55.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836661 1-38654705842' 2026-02-13T02:43:55.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:55.257 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:43:55.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575023 2026-02-13T02:43:55.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575023 2026-02-13T02:43:55.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836661 1-38654705842 2-55834575023' 2026-02-13T02:43:55.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:55.330 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836661 2026-02-13T02:43:55.330 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:55.331 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:43:55.332 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836661 2026-02-13T02:43:55.332 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:55.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836661 2026-02-13T02:43:55.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836661' 2026-02-13T02:43:55.334 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836661 2026-02-13T02:43:55.334 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:55.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836658 -lt 21474836661 2026-02-13T02:43:55.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:43:56.500 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:43:56.501 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:56.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836661 -lt 21474836661 2026-02-13T02:43:56.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:56.662 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705842 2026-02-13T02:43:56.662 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:56.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:43:56.664 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705842 2026-02-13T02:43:56.665 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:56.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705842 2026-02-13T02:43:56.666 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705842 2026-02-13T02:43:56.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705842' 2026-02-13T02:43:56.667 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:43:56.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705842 -lt 38654705842 2026-02-13T02:43:56.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:56.828 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575023 2026-02-13T02:43:56.829 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:56.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:43:56.830 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575023 2026-02-13T02:43:56.830 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:56.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575023 2026-02-13T02:43:56.832 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834575023 2026-02-13T02:43:56.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575023' 2026-02-13T02:43:56.833 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:43:56.998 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575024 -lt 55834575023 2026-02-13T02:43:56.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:43:56.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:43:57.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:43:57.150 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:43:57.160 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:43:57.160 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:43:57.160 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:43:57.160 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:43:57.160 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:43:57.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:43:57.322 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:43:57.322 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:43:57.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:43:57.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:57.322 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:43:57.394 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836664 2026-02-13T02:43:57.394 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836664 2026-02-13T02:43:57.394 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836664' 2026-02-13T02:43:57.394 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:57.394 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:43:57.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705845 2026-02-13T02:43:57.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705845 2026-02-13T02:43:57.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836664 1-38654705845' 2026-02-13T02:43:57.467 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:57.467 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:43:57.537 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575027 2026-02-13T02:43:57.537 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575027 2026-02-13T02:43:57.537 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836664 1-38654705845 2-55834575027' 2026-02-13T02:43:57.537 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:57.538 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836664 2026-02-13T02:43:57.538 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:57.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:43:57.540 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836664 2026-02-13T02:43:57.540 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:57.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836664 2026-02-13T02:43:57.541 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836664 2026-02-13T02:43:57.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836664' 2026-02-13T02:43:57.542 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:57.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836661 -lt 21474836664 2026-02-13T02:43:57.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:43:58.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:43:58.714 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:58.871 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836664 -lt 21474836664 2026-02-13T02:43:58.871 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:58.872 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705845 2026-02-13T02:43:58.872 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:58.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:43:58.874 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705845 2026-02-13T02:43:58.874 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:58.876 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705845 2026-02-13T02:43:58.876 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705845' 2026-02-13T02:43:58.876 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705845 2026-02-13T02:43:58.876 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:43:59.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705845 -lt 38654705845 2026-02-13T02:43:59.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:59.045 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575027 2026-02-13T02:43:59.046 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:59.047 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:43:59.048 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575027 2026-02-13T02:43:59.048 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:59.049 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575027 2026-02-13T02:43:59.049 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575027' 2026-02-13T02:43:59.050 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834575027 2026-02-13T02:43:59.050 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:43:59.219 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575027 -lt 55834575027 2026-02-13T02:43:59.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:43:59.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:43:59.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:43:59.368 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:43:59.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:43:59.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:43:59.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:43:59.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:43:59.379 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:43:59.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:43:59.541 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:43:59.541 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:43:59.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:43:59.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:59.541 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:43:59.615 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836667 2026-02-13T02:43:59.615 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836667 2026-02-13T02:43:59.615 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836667' 2026-02-13T02:43:59.615 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:59.615 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:43:59.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705848 2026-02-13T02:43:59.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705848 2026-02-13T02:43:59.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836667 1-38654705848' 2026-02-13T02:43:59.688 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:43:59.688 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:43:59.760 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575030 2026-02-13T02:43:59.760 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575030 2026-02-13T02:43:59.760 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836667 1-38654705848 2-55834575030' 2026-02-13T02:43:59.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:59.761 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836667 2026-02-13T02:43:59.761 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:59.762 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:43:59.763 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836667 2026-02-13T02:43:59.763 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:59.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836667 2026-02-13T02:43:59.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836667' 2026-02-13T02:43:59.765 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836667 2026-02-13T02:43:59.765 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:43:59.926 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836667 -lt 21474836667 2026-02-13T02:43:59.926 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:43:59.926 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705848 2026-02-13T02:43:59.927 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:43:59.928 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:43:59.929 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705848 2026-02-13T02:43:59.929 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:43:59.930 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705848 2026-02-13T02:43:59.931 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705848 2026-02-13T02:43:59.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705848' 2026-02-13T02:43:59.931 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:44:00.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705848 -lt 38654705848 2026-02-13T02:44:00.092 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:44:00.092 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575030 2026-02-13T02:44:00.093 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:44:00.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:44:00.095 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575030 2026-02-13T02:44:00.095 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:44:00.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575030 2026-02-13T02:44:00.096 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575030' 2026-02-13T02:44:00.097 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834575030 2026-02-13T02:44:00.097 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:44:00.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575029 -lt 55834575030 2026-02-13T02:44:00.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:44:01.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:44:01.262 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:44:01.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575029 -lt 55834575030 2026-02-13T02:44:01.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:44:02.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 499 -eq 0 ']' 2026-02-13T02:44:02.431 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:44:02.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575032 -lt 55834575030 2026-02-13T02:44:02.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:44:02.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:44:02.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:44:02.740 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:44:02.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:44:02.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:44:02.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:44:02.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:44:02.750 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:44:02.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:44:02.911 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:44:02.911 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:44:02.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:44:02.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:44:02.911 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:44:02.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836672 2026-02-13T02:44:02.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836672 2026-02-13T02:44:02.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836672' 2026-02-13T02:44:02.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:44:02.984 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:44:03.057 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705853 2026-02-13T02:44:03.057 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705853 2026-02-13T02:44:03.057 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836672 1-38654705853' 2026-02-13T02:44:03.057 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:44:03.057 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:44:03.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575034 2026-02-13T02:44:03.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575034 2026-02-13T02:44:03.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836672 1-38654705853 2-55834575034' 2026-02-13T02:44:03.131 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:44:03.132 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836672 2026-02-13T02:44:03.132 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:44:03.133 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:44:03.134 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836672 2026-02-13T02:44:03.134 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:44:03.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836672 2026-02-13T02:44:03.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836672' 2026-02-13T02:44:03.135 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836672 2026-02-13T02:44:03.136 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:44:03.297 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836669 -lt 21474836672 2026-02-13T02:44:03.298 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:44:04.299 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:44:04.299 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:44:04.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836672 -lt 21474836672 2026-02-13T02:44:04.460 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:44:04.461 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705853 2026-02-13T02:44:04.461 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:44:04.462 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:44:04.463 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705853 2026-02-13T02:44:04.463 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:44:04.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705853 2026-02-13T02:44:04.465 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705853 2026-02-13T02:44:04.465 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705853' 2026-02-13T02:44:04.465 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:44:04.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705853 -lt 38654705853 2026-02-13T02:44:04.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:44:04.639 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575034 2026-02-13T02:44:04.639 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:44:04.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:44:04.641 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575034 2026-02-13T02:44:04.641 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:44:04.642 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575034 2026-02-13T02:44:04.642 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834575034 2026-02-13T02:44:04.643 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575034' 2026-02-13T02:44:04.643 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:44:04.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575035 -lt 55834575034 2026-02-13T02:44:04.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:44:04.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:44:04.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:44:04.959 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:44:04.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:44:04.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:44:04.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:44:04.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:44:04.969 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:44:05.128 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:44:05.128 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:44:05.129 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:44:05.129 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:44:05.129 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:44:05.129 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:44:05.201 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836675 2026-02-13T02:44:05.201 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836675 2026-02-13T02:44:05.201 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836675' 2026-02-13T02:44:05.201 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:44:05.201 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:44:05.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705856 2026-02-13T02:44:05.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705856 2026-02-13T02:44:05.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836675 1-38654705856' 2026-02-13T02:44:05.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:44:05.273 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:44:05.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575037 2026-02-13T02:44:05.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575037 2026-02-13T02:44:05.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836675 1-38654705856 2-55834575037' 2026-02-13T02:44:05.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:44:05.345 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836675 2026-02-13T02:44:05.345 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:44:05.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:44:05.347 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836675 2026-02-13T02:44:05.347 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:44:05.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836675 2026-02-13T02:44:05.349 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836675' 2026-02-13T02:44:05.349 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836675 2026-02-13T02:44:05.349 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:44:05.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836672 -lt 21474836675 2026-02-13T02:44:05.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:44:06.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:44:06.523 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:44:06.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836675 -lt 21474836675 2026-02-13T02:44:06.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:44:06.685 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705856 2026-02-13T02:44:06.686 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:44:06.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:44:06.687 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705856 2026-02-13T02:44:06.688 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:44:06.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705856 2026-02-13T02:44:06.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705856' 2026-02-13T02:44:06.689 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705856 2026-02-13T02:44:06.689 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:44:06.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705856 -lt 38654705856 2026-02-13T02:44:06.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:44:06.856 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575037 2026-02-13T02:44:06.856 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:44:06.857 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:44:06.858 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575037 2026-02-13T02:44:06.858 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:44:06.859 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575037 2026-02-13T02:44:06.859 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575037' 2026-02-13T02:44:06.860 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834575037 2026-02-13T02:44:06.860 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:44:07.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575038 -lt 55834575037 2026-02-13T02:44:07.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:44:07.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:44:07.034 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:44:07.181 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:44:07.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:44:07.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:44:07.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:44:07.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:44:07.191 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:44:07.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:44:07.346 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:44:07.346 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:44:07.347 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:44:07.347 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:44:07.347 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:44:07.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836678 2026-02-13T02:44:07.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836678 2026-02-13T02:44:07.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836678' 2026-02-13T02:44:07.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:44:07.424 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:44:07.497 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705859 2026-02-13T02:44:07.497 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705859 2026-02-13T02:44:07.497 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836678 1-38654705859' 2026-02-13T02:44:07.497 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:44:07.498 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:44:07.568 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575041 2026-02-13T02:44:07.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575041 2026-02-13T02:44:07.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836678 1-38654705859 2-55834575041' 2026-02-13T02:44:07.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:44:07.570 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836678 2026-02-13T02:44:07.570 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:44:07.570 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:44:07.571 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836678 2026-02-13T02:44:07.571 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:44:07.572 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836678 2026-02-13T02:44:07.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836678' 2026-02-13T02:44:07.573 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836678 2026-02-13T02:44:07.573 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:44:07.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836678 -lt 21474836678 2026-02-13T02:44:07.744 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:44:07.745 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705859 2026-02-13T02:44:07.745 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:44:07.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:44:07.747 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705859 2026-02-13T02:44:07.747 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:44:07.748 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705859 2026-02-13T02:44:07.748 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705859 2026-02-13T02:44:07.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705859' 2026-02-13T02:44:07.749 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:44:07.906 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705859 -lt 38654705859 2026-02-13T02:44:07.906 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:44:07.906 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575041 2026-02-13T02:44:07.907 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:44:07.907 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:44:07.908 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575041 2026-02-13T02:44:07.909 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:44:07.910 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575041 2026-02-13T02:44:07.910 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834575041 2026-02-13T02:44:07.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575041' 2026-02-13T02:44:07.911 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:44:08.072 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575041 -lt 55834575041 2026-02-13T02:44:08.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:44:08.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:44:08.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:44:08.224 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:44:08.233 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:44:08.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:44:08.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:44:08.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:44:08.234 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:44:08.394 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:44:08.394 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:44:08.394 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:44:08.394 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:44:08.394 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:44:08.394 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:44:08.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836680 2026-02-13T02:44:08.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836680 2026-02-13T02:44:08.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836680' 2026-02-13T02:44:08.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:44:08.466 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:44:08.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705861 2026-02-13T02:44:08.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705861 2026-02-13T02:44:08.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836680 1-38654705861' 2026-02-13T02:44:08.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:44:08.546 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:44:08.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575043 2026-02-13T02:44:08.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575043 2026-02-13T02:44:08.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836680 1-38654705861 2-55834575043' 2026-02-13T02:44:08.617 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:44:08.618 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836680 2026-02-13T02:44:08.618 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:44:08.619 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:44:08.619 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836680 2026-02-13T02:44:08.619 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:44:08.621 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836680 2026-02-13T02:44:08.621 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836680' 2026-02-13T02:44:08.621 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836680 2026-02-13T02:44:08.622 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:44:08.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836678 -lt 21474836680 2026-02-13T02:44:08.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:44:09.791 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:44:09.792 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:44:09.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836681 -lt 21474836680 2026-02-13T02:44:09.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:44:09.957 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705861 2026-02-13T02:44:09.957 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:44:09.958 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:44:09.959 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705861 2026-02-13T02:44:09.959 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:44:09.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705861 2026-02-13T02:44:09.960 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705861' 2026-02-13T02:44:09.961 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705861 2026-02-13T02:44:09.961 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:44:10.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705862 -lt 38654705861 2026-02-13T02:44:10.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:44:10.120 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575043 2026-02-13T02:44:10.120 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:44:10.121 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:44:10.122 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575043 2026-02-13T02:44:10.122 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:44:10.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575043 2026-02-13T02:44:10.124 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834575043 2026-02-13T02:44:10.124 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575043' 2026-02-13T02:44:10.124 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:44:10.295 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575044 -lt 55834575043 2026-02-13T02:44:10.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:44:10.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:44:10.296 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:44:10.443 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:44:10.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:44:10.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:44:10.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:44:10.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:44:10.454 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:44:10.610 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:44:10.611 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:44:10.611 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:44:10.611 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:44:10.611 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:44:10.611 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:44:10.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836683 2026-02-13T02:44:10.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836683 2026-02-13T02:44:10.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836683' 2026-02-13T02:44:10.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:44:10.686 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:44:10.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705864 2026-02-13T02:44:10.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705864 2026-02-13T02:44:10.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836683 1-38654705864' 2026-02-13T02:44:10.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:44:10.756 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:44:10.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575046 2026-02-13T02:44:10.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575046 2026-02-13T02:44:10.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836683 1-38654705864 2-55834575046' 2026-02-13T02:44:10.828 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:44:10.829 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836683 2026-02-13T02:44:10.829 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:44:10.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:44:10.831 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836683 2026-02-13T02:44:10.831 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:44:10.832 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836683 2026-02-13T02:44:10.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836683' 2026-02-13T02:44:10.833 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836683 2026-02-13T02:44:10.833 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:44:11.004 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836681 -lt 21474836683 2026-02-13T02:44:11.004 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:44:12.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:44:12.006 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:44:12.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836684 -lt 21474836683 2026-02-13T02:44:12.166 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:44:12.167 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705864 2026-02-13T02:44:12.167 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:44:12.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:44:12.169 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705864 2026-02-13T02:44:12.169 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:44:12.170 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705864 2026-02-13T02:44:12.170 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705864' 2026-02-13T02:44:12.171 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705864 2026-02-13T02:44:12.171 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:44:12.333 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705865 -lt 38654705864 2026-02-13T02:44:12.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:44:12.334 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575046 2026-02-13T02:44:12.334 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:44:12.336 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:44:12.336 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575046 2026-02-13T02:44:12.337 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:44:12.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575046 2026-02-13T02:44:12.338 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575046' 2026-02-13T02:44:12.338 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834575046 2026-02-13T02:44:12.339 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:44:12.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575047 -lt 55834575046 2026-02-13T02:44:12.505 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:44:12.505 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:44:12.506 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:44:12.655 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:44:12.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-13T02:44:12.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-13T02:44:12.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-13T02:44:12.667 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:44:12.667 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:44:12.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:44:12.829 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:44:12.829 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:44:12.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:44:12.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:44:12.829 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:44:12.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836687 2026-02-13T02:44:12.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836687 2026-02-13T02:44:12.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836687' 2026-02-13T02:44:12.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:44:12.902 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:44:12.975 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705868 2026-02-13T02:44:12.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705868 2026-02-13T02:44:12.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836687 1-38654705868' 2026-02-13T02:44:12.976 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:44:12.976 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:44:13.047 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575049 2026-02-13T02:44:13.048 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575049 2026-02-13T02:44:13.048 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836687 1-38654705868 2-55834575049' 2026-02-13T02:44:13.048 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:44:13.048 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836687 2026-02-13T02:44:13.048 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:44:13.049 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:44:13.050 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836687 2026-02-13T02:44:13.050 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:44:13.052 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836687 2026-02-13T02:44:13.052 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836687 2026-02-13T02:44:13.052 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836687' 2026-02-13T02:44:13.053 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:44:13.224 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836684 -lt 21474836687 2026-02-13T02:44:13.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:44:14.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:44:14.227 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:44:14.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836687 -lt 21474836687 2026-02-13T02:44:14.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:44:14.391 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705868 2026-02-13T02:44:14.391 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:44:14.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:44:14.393 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705868 2026-02-13T02:44:14.393 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:44:14.395 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705868 2026-02-13T02:44:14.395 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705868' 2026-02-13T02:44:14.395 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 38654705868 2026-02-13T02:44:14.395 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:44:14.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705868 -lt 38654705868 2026-02-13T02:44:14.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:44:14.555 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575049 2026-02-13T02:44:14.555 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:44:14.556 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:44:14.557 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575049 2026-02-13T02:44:14.557 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:44:14.558 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575049 2026-02-13T02:44:14.559 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 55834575049 2026-02-13T02:44:14.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575049' 2026-02-13T02:44:14.559 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:44:14.720 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575050 -lt 55834575049 2026-02-13T02:44:14.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-13T02:44:14.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-13T02:44:14.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-13T02:44:14.871 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:44:14.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: break 2026-02-13T02:44:14.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: set +o pipefail 2026-02-13T02:44:14.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: sleep 5 2026-02-13T02:44:19.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:384: _scrub_abort: grep 'noscrub set, aborting' td/osd-scrub-test/osd.1.log 2026-02-13T02:44:19.892 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T02:42:17.646+0000 7fd800df9640 10 osd.1 pg_epoch: 18 pg[1.0( v 17'1000 (0'0,17'1000] local-lis/les=14/15 n=1000 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0,2] r=0 lpr=14 crt=17'1000 lcod 17'999 mlcod 17'999 active+clean+scrubbing(periodic-regular)] scrubber: noscrub set, aborting 2026-02-13T02:44:19.893 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:390: _scrub_abort: get_last_scrub_stamp 1.0 2026-02-13T02:44:19.893 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:44:19.893 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:44:19.893 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:44:19.893 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:44:20.052 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:390: _scrub_abort: local last_scrub=2026-02-11T02:42:12.483336+0000 2026-02-13T02:44:20.052 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:391: _scrub_abort: ceph config set osd osd_scrub_sleep 0.1 2026-02-13T02:44:20.219 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:393: _scrub_abort: ceph osd unset noscrub 2026-02-13T02:44:20.411 INFO:tasks.workunit.client.0.trial112.stderr:noscrub is unset 2026-02-13T02:44:20.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:394: _scrub_abort: '[' scrub = deep-scrub ']' 2026-02-13T02:44:20.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:398: _scrub_abort: TIMEOUT=500 2026-02-13T02:44:20.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:399: _scrub_abort: wait_for_scrub 1.0 2026-02-11T02:42:12.483336+0000 2026-02-13T02:44:20.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-13T02:44:20.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-11T02:42:12.483336+0000 2026-02-13T02:44:20.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:44:20.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:44:20.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:44:20.424 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:44:20.424 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:44:20.424 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:44:20.424 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:44:20.425 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:44:20.585 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:44:20.585 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:44:21.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:44:21.587 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:44:21.588 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:44:21.588 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:44:21.588 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:44:21.588 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:44:21.588 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:44:21.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:44:21.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:44:22.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:44:22.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:44:22.751 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:44:22.752 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:44:22.752 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:44:22.752 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:44:22.752 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:44:22.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:44:22.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:44:23.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:44:23.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:44:23.913 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:44:23.914 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:44:23.914 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:44:23.914 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:44:23.914 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:44:24.072 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:44:24.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:44:25.074 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:44:25.074 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:44:25.075 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:44:25.075 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:44:25.075 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:44:25.076 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:44:25.076 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:44:25.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:44:25.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:44:26.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:44:26.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:44:26.236 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:44:26.236 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:44:26.237 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:44:26.237 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:44:26.237 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:44:26.395 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:44:26.395 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:44:27.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:44:27.396 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:44:27.397 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:44:27.397 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:44:27.397 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:44:27.397 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:44:27.397 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:44:27.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:44:27.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:44:28.556 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:44:28.556 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:44:28.557 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:44:28.557 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:44:28.557 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:44:28.558 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:44:28.558 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:44:28.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:44:28.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:44:29.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:44:29.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:44:29.716 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:44:29.716 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:44:29.716 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:44:29.717 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:44:29.717 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:44:29.878 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:44:29.878 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:44:30.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:44:30.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:44:30.881 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:44:30.881 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:44:30.881 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:44:30.881 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:44:30.881 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:44:31.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:44:31.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:44:32.042 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:44:32.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:44:32.044 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:44:32.044 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:44:32.044 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:44:32.044 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:44:32.044 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:44:32.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:44:32.207 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:44:33.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:44:33.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:44:33.209 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:44:33.209 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:44:33.209 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:44:33.210 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:44:33.210 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:44:33.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:44:33.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:44:34.368 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:44:34.368 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:44:34.369 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:44:34.369 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:44:34.370 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:44:34.370 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:44:34.370 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:44:34.527 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:44:34.527 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:44:35.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:44:35.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:44:35.529 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:44:35.529 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:44:35.530 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:44:35.530 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:44:35.530 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:44:35.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:44:35.686 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:44:36.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:44:36.687 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:44:36.688 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:44:36.688 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:44:36.688 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:44:36.689 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:44:36.689 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:44:36.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:44:36.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:44:37.847 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:44:37.847 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:44:37.848 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:44:37.848 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:44:37.848 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:44:37.848 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:44:37.849 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:44:38.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:44:38.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:44:39.004 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:44:39.004 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:44:39.005 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:44:39.005 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:44:39.005 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:44:39.006 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:44:39.006 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:44:39.160 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:44:39.160 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:44:40.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:44:40.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:44:40.162 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:44:40.162 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:44:40.162 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:44:40.162 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:44:40.163 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:44:40.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:44:40.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:44:41.319 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:44:41.319 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:44:41.320 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:44:41.320 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:44:41.320 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:44:41.321 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:44:41.321 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:44:41.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:44:41.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:44:42.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:44:42.478 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:44:42.478 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:44:42.478 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:44:42.479 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:44:42.479 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:44:42.479 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:44:42.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:44:42.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:44:43.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:44:43.635 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:44:43.636 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:44:43.636 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:44:43.636 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:44:43.636 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:44:43.636 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:44:43.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:44:43.792 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:44:44.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:44:44.793 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:44:44.794 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:44:44.794 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:44:44.794 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:44:44.795 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:44:44.795 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:44:44.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:44:44.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:44:45.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:44:45.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:44:45.954 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:44:45.954 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:44:45.954 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:44:45.955 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:44:45.955 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:44:46.110 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:44:46.111 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:44:47.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:44:47.112 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:44:47.113 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:44:47.113 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:44:47.113 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:44:47.113 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:44:47.113 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:44:47.271 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:44:47.271 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:44:48.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:44:48.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:44:48.273 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:44:48.273 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:44:48.273 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:44:48.273 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:44:48.273 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:44:48.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:44:48.426 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:44:49.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:44:49.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:44:49.428 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:44:49.429 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:44:49.429 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:44:49.429 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:44:49.429 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:44:49.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:44:49.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:44:50.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:44:50.594 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:44:50.595 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:44:50.595 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:44:50.595 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:44:50.595 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:44:50.595 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:44:50.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:44:50.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:44:51.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:44:51.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:44:51.760 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:44:51.760 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:44:51.760 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:44:51.760 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:44:51.760 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:44:51.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:44:51.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:44:52.924 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:44:52.924 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:44:52.925 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:44:52.926 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:44:52.926 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:44:52.926 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:44:52.926 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:44:53.086 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:44:53.086 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:44:54.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:44:54.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:44:54.089 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:44:54.089 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:44:54.089 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:44:54.089 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:44:54.089 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:44:54.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:44:54.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:44:55.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:44:55.251 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:44:55.252 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:44:55.252 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:44:55.252 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:44:55.252 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:44:55.252 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:44:55.410 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:44:55.410 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:44:56.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:44:56.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:44:56.413 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:44:56.413 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:44:56.413 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:44:56.413 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:44:56.413 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:44:56.572 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:44:56.572 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:44:57.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:44:57.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:44:57.574 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:44:57.575 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:44:57.575 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:44:57.575 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:44:57.575 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:44:57.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:44:57.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:44:58.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:44:58.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:44:58.734 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:44:58.734 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:44:58.734 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:44:58.734 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:44:58.734 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:44:58.893 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:44:58.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:44:59.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:44:59.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:44:59.895 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:44:59.895 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:44:59.895 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:44:59.896 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:44:59.896 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:00.058 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:00.058 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:01.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:01.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:01.061 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:01.061 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:01.061 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:01.061 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:01.061 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:01.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:01.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:02.222 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:02.222 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:02.222 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:02.222 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:02.222 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:02.223 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:02.223 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:02.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:02.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:03.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:03.387 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:03.388 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:03.389 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:03.389 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:03.389 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:03.389 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:03.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:03.550 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:04.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:04.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:04.552 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:04.552 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:04.552 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:04.552 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:04.553 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:04.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:04.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:05.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:05.713 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:05.714 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:05.714 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:05.714 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:05.715 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:05.715 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:05.877 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:05.878 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:06.878 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:06.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:06.880 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:06.880 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:06.880 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:06.880 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:06.880 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:07.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:07.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:08.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:08.039 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:08.040 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:08.040 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:08.041 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:08.041 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:08.041 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:08.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:08.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:09.198 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:09.198 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:09.199 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:09.199 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:09.199 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:09.199 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:09.200 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:09.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:09.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:10.361 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:10.361 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:10.362 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:10.362 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:10.362 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:10.362 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:10.362 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:10.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:10.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:11.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:11.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:11.523 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:11.523 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:11.523 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:11.523 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:11.524 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:11.681 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:11.681 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:12.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:12.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:12.683 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:12.683 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:12.683 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:12.683 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:12.683 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:12.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:12.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:13.845 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:13.845 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:13.846 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:13.846 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:13.846 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:13.846 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:13.847 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:14.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:14.004 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:15.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:15.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:15.005 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:15.006 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:15.006 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:15.006 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:15.006 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:15.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:15.163 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:16.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:16.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:16.165 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:16.165 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:16.165 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:16.165 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:16.165 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:16.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:16.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:17.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:17.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:17.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:17.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:17.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:17.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:17.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:17.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:17.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:18.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:18.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:18.490 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:18.490 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:18.490 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:18.491 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:18.491 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:18.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:18.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:19.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:19.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:19.653 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:19.653 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:19.654 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:19.654 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:19.654 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:19.812 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:19.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:20.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:20.814 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:20.814 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:20.814 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:20.815 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:20.815 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:20.815 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:20.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:20.978 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:21.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:21.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:21.980 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:21.980 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:21.981 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:21.981 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:21.981 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:22.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:22.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:23.143 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:23.144 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:23.145 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:23.145 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:23.145 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:23.145 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:23.145 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:23.303 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:23.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:24.304 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:24.305 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:24.306 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:24.306 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:24.306 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:24.306 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:24.306 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:24.463 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:24.463 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:25.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:25.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:25.465 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:25.465 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:25.466 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:25.466 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:25.466 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:25.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:25.623 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:26.624 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:26.625 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:26.626 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:26.626 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:26.626 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:26.626 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:26.626 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:26.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:26.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:27.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:27.785 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:27.785 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:27.786 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:27.786 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:27.786 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:27.786 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:27.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:27.944 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:28.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:28.945 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:28.946 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:28.946 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:28.947 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:28.947 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:28.947 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:29.105 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:29.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:30.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:30.107 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:30.108 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:30.108 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:30.108 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:30.108 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:30.108 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:30.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:30.266 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:31.266 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:31.267 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:31.268 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:31.268 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:31.268 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:31.268 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:31.268 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:31.427 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:31.428 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:32.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:32.429 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:32.430 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:32.430 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:32.430 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:32.430 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:32.430 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:32.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:32.589 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:33.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:33.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:33.591 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:33.591 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:33.591 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:33.591 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:33.591 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:33.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:33.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:34.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:34.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:34.752 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:34.752 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:34.752 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:34.752 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:34.752 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:34.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:34.912 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:35.913 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:35.913 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:35.914 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:35.914 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:35.914 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:35.914 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:35.914 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:36.072 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:36.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:37.074 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:37.074 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:37.075 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:37.075 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:37.075 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:37.075 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:37.075 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:37.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:37.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:38.237 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:38.238 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:38.238 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:38.238 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:38.239 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:38.239 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:38.239 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:38.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:38.399 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:39.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:39.400 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:39.401 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:39.401 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:39.401 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:39.402 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:39.402 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:39.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:39.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:40.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:40.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:40.563 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:40.563 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:40.563 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:40.563 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:40.563 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:40.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:40.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:41.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:41.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:41.724 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:41.724 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:41.724 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:41.724 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:41.724 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:41.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:41.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:42.886 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:42.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:42.887 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:42.887 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:42.887 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:42.888 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:42.888 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:43.050 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:43.050 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:44.051 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:44.051 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:44.052 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:44.052 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:44.052 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:44.053 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:44.053 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:44.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:44.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:45.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:45.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:45.214 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:45.214 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:45.214 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:45.215 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:45.215 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:45.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:45.375 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:46.376 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:46.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:46.377 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:46.377 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:46.377 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:46.378 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:46.378 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:46.537 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:46.537 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:47.538 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:47.539 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:47.539 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:47.540 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:47.540 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:47.540 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:47.540 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:47.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-11T02:42:12.483336+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:47.698 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:45:48.699 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:45:48.700 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:45:48.701 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:45:48.701 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:45:48.701 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:45:48.701 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:45:48.701 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:45:48.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:45:46.303322+0000 '>' 2026-02-11T02:42:12.483336+0000 2026-02-13T02:45:48.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:45:48.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:400: _scrub_abort: perf_counters td/osd-scrub-test 3 2026-02-13T02:45:48.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2026-02-13T02:45:48.861 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=3 2026-02-13T02:45:48.861 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 3 - 1 2026-02-13T02:45:48.862 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 2 2026-02-13T02:45:48.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:45:48.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2026-02-13T02:45:48.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-13T02:45:48.943 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:45:48.944 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_ec": [ 2026-02-13T02:45:48.944 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:45:48.944 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:45:48.944 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:45:48.944 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:45:48.944 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:45:48.944 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:45:48.944 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:45:48.944 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:45:48.945 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:45:48.945 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:45:48.945 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:45:48.945 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:45:48.945 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:45:48.945 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:45:48.945 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_repl": [ 2026-02-13T02:45:48.945 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:45:48.945 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:45:48.945 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:45:48.945 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:45:48.945 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:45:48.946 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:45:48.946 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:45:48.946 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:45:48.946 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:45:48.946 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:45:48.946 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:45:48.946 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:45:48.946 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:45:48.946 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:45:48.946 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_ec": [ 2026-02-13T02:45:48.946 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:45:48.946 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:45:48.946 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:45:48.947 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:45:48.947 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:45:48.947 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:45:48.947 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:45:48.947 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:45:48.947 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:45:48.947 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:45:48.947 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:45:48.947 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:45:48.947 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:45:48.947 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:45:48.947 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_repl": [ 2026-02-13T02:45:48.947 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:45:48.948 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:45:48.948 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:45:48.948 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:45:48.948 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:45:48.948 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:45:48.948 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:45:48.948 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:45:48.948 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:45:48.948 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:45:48.948 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:45:48.948 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:45:48.948 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:45:48.948 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:45:48.949 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:45:48.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:45:48.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2026-02-13T02:45:48.950 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-13T02:45:49.013 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:45:49.013 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_ec": [ 2026-02-13T02:45:49.013 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:45:49.013 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:45:49.014 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:45:49.014 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:45:49.014 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:45:49.014 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:45:49.014 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:45:49.014 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:45:49.014 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:45:49.014 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:45:49.014 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:45:49.014 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:45:49.014 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:45:49.014 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:45:49.014 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_repl": [ 2026-02-13T02:45:49.015 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:45:49.015 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:45:49.015 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:45:49.015 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:45:49.015 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:45:49.015 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:45:49.015 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:45:49.015 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:45:49.015 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:45:49.015 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:45:49.015 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:45:49.015 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:45:49.015 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:45:49.016 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:45:49.016 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_ec": [ 2026-02-13T02:45:49.016 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:45:49.016 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:45:49.016 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:45:49.016 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:45:49.016 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:45:49.016 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:45:49.016 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:45:49.016 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:45:49.016 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:45:49.016 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:45:49.016 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:45:49.017 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:45:49.017 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:45:49.017 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:45:49.017 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_repl": [ 2026-02-13T02:45:49.017 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:45:49.017 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:45:49.017 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:45:49.017 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:45:49.017 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:45:49.017 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:45:49.017 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:45:49.017 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 11, 2026-02-13T02:45:49.017 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:45:49.018 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:45:49.018 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:45:49.018 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:45:49.018 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:45:49.018 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:45:49.018 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:45:49.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:45:49.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.2 counter dump 2026-02-13T02:45:49.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-13T02:45:49.083 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:45:49.084 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_ec": [ 2026-02-13T02:45:49.084 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:45:49.084 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:45:49.084 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:45:49.084 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:45:49.084 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:45:49.084 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:45:49.084 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:45:49.084 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:45:49.084 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:45:49.084 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:45:49.084 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:45:49.085 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:45:49.085 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:45:49.085 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:45:49.085 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_repl": [ 2026-02-13T02:45:49.085 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:45:49.085 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:45:49.085 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:45:49.085 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:45:49.085 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:45:49.085 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:45:49.085 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:45:49.085 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:45:49.085 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:45:49.086 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:45:49.086 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:45:49.086 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:45:49.086 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:45:49.086 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:45:49.086 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_ec": [ 2026-02-13T02:45:49.086 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:45:49.086 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:45:49.086 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:45:49.086 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:45:49.086 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:45:49.086 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:45:49.087 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:45:49.087 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:45:49.087 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:45:49.087 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:45:49.087 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:45:49.087 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:45:49.087 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:45:49.087 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:45:49.087 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_repl": [ 2026-02-13T02:45:49.087 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:45:49.087 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:45:49.087 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:45:49.087 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:45:49.088 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:45:49.088 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:45:49.088 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:45:49.088 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:45:49.088 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:45:49.088 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:45:49.088 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:45:49.088 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:45:49.088 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:45:49.088 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:45:49.088 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:45:49.089 INFO:tasks.workunit.client.0.trial112.stdout:-------------- Teardown Test TEST_scrub_abort ------------------ 2026-02-13T02:45:49.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_scrub_abort ------------------' 2026-02-13T02:45:49.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2026-02-13T02:45:49.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-13T02:45:49.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:45:49.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-13T02:45:49.090 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:45:49.090 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:45:49.090 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:45:49.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:45:49.091 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:45:49.194 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:45:49.195 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:45:49.196 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:45:49.196 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:45:49.198 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:45:49.198 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:45:49.198 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:45:49.199 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:45:49.199 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:45:49.200 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:45:49.200 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:45:49.201 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:45:49.202 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:45:49.202 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-13T02:45:49.252 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:45:49.252 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:45:49.252 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:45:49.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.508946 2026-02-13T02:45:49.254 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:45:49.254 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:45:49.254 INFO:tasks.workunit.client.0.trial112.stdout:-------------- Complete Test TEST_scrub_abort ------------------ 2026-02-13T02:45:49.254 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_scrub_abort ------------------' 2026-02-13T02:45:49.254 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2026-02-13T02:45:49.254 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_scrub_permit_time -------------------' 2026-02-13T02:45:49.254 INFO:tasks.workunit.client.0.trial112.stdout:-------------- Prepare Test TEST_scrub_permit_time ------------------- 2026-02-13T02:45:49.254 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2026-02-13T02:45:49.254 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2026-02-13T02:45:49.254 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2026-02-13T02:45:49.254 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-13T02:45:49.254 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:45:49.254 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-13T02:45:49.254 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:45:49.254 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:45:49.255 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:45:49.255 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:45:49.255 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:45:49.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:45:49.258 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:45:49.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:45:49.259 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:45:49.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:45:49.260 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:45:49.261 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:45:49.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:45:49.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:45:49.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:45:49.263 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:45:49.264 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:45:49.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:45:49.265 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-13T02:45:49.266 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:45:49.266 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:45:49.266 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:45:49.267 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.508946 2026-02-13T02:45:49.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:45:49.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:45:49.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2026-02-13T02:45:49.270 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T02:45:49.270 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:45:49.270 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:45:49.270 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.508946 2026-02-13T02:45:49.272 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T02:45:49.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-13T02:45:49.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T02:45:49.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2026-02-13T02:45:49.272 INFO:tasks.workunit.client.0.trial112.stdout:-------------- Run Test TEST_scrub_permit_time ----------------------- 2026-02-13T02:45:49.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_scrub_permit_time -----------------------' 2026-02-13T02:45:49.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_scrub_permit_time td/osd-scrub-test 2026-02-13T02:45:49.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:414: TEST_scrub_permit_time: local dir=td/osd-scrub-test 2026-02-13T02:45:49.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:415: TEST_scrub_permit_time: local poolname=test 2026-02-13T02:45:49.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:416: TEST_scrub_permit_time: local OSDS=3 2026-02-13T02:45:49.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:417: TEST_scrub_permit_time: local objects=15 2026-02-13T02:45:49.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:419: TEST_scrub_permit_time: TESTDATA=testdata.508946 2026-02-13T02:45:49.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:421: TEST_scrub_permit_time: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2026-02-13T02:45:49.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-test 2026-02-13T02:45:49.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-13T02:45:49.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-13T02:45:49.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-13T02:45:49.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-test/a 2026-02-13T02:45:49.273 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2026-02-13T02:45:49.293 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-13T02:45:49.293 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:45:49.293 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:45:49.293 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:45:49.293 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:45:49.294 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:45:49.294 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:45:49.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2026-02-13T02:45:49.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-13T02:45:49.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-13T02:45:49.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:45:49.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:45:49.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-13T02:45:49.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-13T02:45:49.324 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:45:49.324 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:45:49.324 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:45:49.324 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:45:49.324 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:45:49.324 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:45:49.325 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.508946/ceph-mon.a.asok 2026-02-13T02:45:49.325 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:45:49.325 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.508946/ceph-mon.a.asok config get fsid 2026-02-13T02:45:49.379 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-13T02:45:49.379 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:45:49.379 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:45:49.379 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-13T02:45:49.380 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-13T02:45:49.380 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:45:49.380 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:45:49.380 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:45:49.381 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:45:49.381 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:45:49.381 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:45:49.381 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.508946/ceph-mon.a.asok 2026-02-13T02:45:49.381 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:45:49.381 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.508946/ceph-mon.a.asok config get mon_host 2026-02-13T02:45:49.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:422: TEST_scrub_permit_time: run_mgr td/osd-scrub-test x --mgr_stats_period=1 2026-02-13T02:45:49.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-test 2026-02-13T02:45:49.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-13T02:45:49.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-13T02:45:49.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-13T02:45:49.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-test/x 2026-02-13T02:45:49.436 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-13T02:45:49.531 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-13T02:45:49.531 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:45:49.531 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:45:49.531 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:45:49.531 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:45:49.531 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:45:49.532 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:45:49.533 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:45:49.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mgr_stats_period=1 2026-02-13T02:45:49.552 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:423: TEST_scrub_permit_time: date -d '2 hour ago' +%H 2026-02-13T02:45:49.553 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:423: TEST_scrub_permit_time: sed 's/^0//' 2026-02-13T02:45:49.554 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:423: TEST_scrub_permit_time: local scrub_begin_hour=0 2026-02-13T02:45:49.555 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:424: TEST_scrub_permit_time: date -d '1 hour ago' +%H 2026-02-13T02:45:49.555 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:424: TEST_scrub_permit_time: sed 's/^0//' 2026-02-13T02:45:49.556 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:424: TEST_scrub_permit_time: local scrub_end_hour=1 2026-02-13T02:45:49.557 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:425: TEST_scrub_permit_time: expr 3 - 1 2026-02-13T02:45:49.558 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:425: TEST_scrub_permit_time: seq 0 2 2026-02-13T02:45:49.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:425: TEST_scrub_permit_time: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:45:49.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:427: TEST_scrub_permit_time: run_osd td/osd-scrub-test 0 --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=0 --osd_scrub_end_hour=1 2026-02-13T02:45:49.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-13T02:45:49.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:45:49.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-13T02:45:49.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:45:49.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/0 2026-02-13T02:45:49.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-13T02:45:49.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:45:49.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:45:49.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:45:49.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2026-02-13T02:45:49.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2026-02-13T02:45:49.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:45:49.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:45:49.560 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-13T02:45:49.560 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:45:49.560 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:45:49.560 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:45:49.560 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:45:49.560 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:45:49.560 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:45:49.560 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:45:49.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:45:49.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:45:49.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:45:49.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:45:49.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-13T02:45:49.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-13T02:45:49.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:45:49.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:45:49.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:45:49.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:45:49.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:45:49.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=0 --osd_scrub_end_hour=1' 2026-02-13T02:45:49.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/0 2026-02-13T02:45:49.562 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:45:49.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2331e813-4a0e-403c-96d2-3ed8bc19bc4c 2026-02-13T02:45:49.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 2331e813-4a0e-403c-96d2-3ed8bc19bc4c' 2026-02-13T02:45:49.563 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 2331e813-4a0e-403c-96d2-3ed8bc19bc4c 2026-02-13T02:45:49.563 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:45:49.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBdkI5plsbzIRAAE3Q6IcTGX8UI0RIiCpuraw== 2026-02-13T02:45:49.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBdkI5plsbzIRAAE3Q6IcTGX8UI0RIiCpuraw=="}' 2026-02-13T02:45:49.573 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2331e813-4a0e-403c-96d2-3ed8bc19bc4c -i td/osd-scrub-test/0/new.json 2026-02-13T02:45:49.663 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:45:49.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/0/new.json 2026-02-13T02:45:49.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=0 --osd_scrub_end_hour=1 --mkfs --key AQBdkI5plsbzIRAAE3Q6IcTGX8UI0RIiCpuraw== --osd-uuid 2331e813-4a0e-403c-96d2-3ed8bc19bc4c 2026-02-13T02:45:49.688 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:45:49.680+0000 7f8e002c9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:45:49.690 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:45:49.684+0000 7f8e002c9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:45:49.691 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:45:49.684+0000 7f8e002c9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:45:49.691 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:45:49.684+0000 7f8e002c9a40 -1 bdev(0x5609d4f38c00 td/osd-scrub-test/0/block) open stat got: (1) Operation not permitted 2026-02-13T02:45:49.691 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:45:49.684+0000 7f8e002c9a40 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2026-02-13T02:45:50.463 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2026-02-13T02:45:50.463 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:45:50.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-13T02:45:50.464 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T02:45:50.464 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:45:50.568 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T02:45:50.568 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-13T02:45:50.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=0 --osd_scrub_end_hour=1 2026-02-13T02:45:50.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:45:50.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:45:50.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:45:50.595 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:45:50.588+0000 7f73887e8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:45:50.597 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:45:50.588+0000 7f73887e8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:45:50.598 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:45:50.592+0000 7f73887e8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:45:50.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-13T02:45:50.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:45:50.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:45:50.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:45:50.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:45:50.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:45:50.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:45:50.658 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:45:50.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:45:50.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:45:50.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:45:50.975 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:45:50.968+0000 7f73887e8a40 -1 Falling back to public interface 2026-02-13T02:45:51.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:45:51.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:45:51.750 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:45:51.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:45:51.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:45:51.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:45:51.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:45:51.984 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:45:51.976+0000 7f73887e8a40 -1 osd.0 0 log_to_monitors true 2026-02-13T02:45:52.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:45:52.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:45:52.844 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:45:52.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:45:52.844 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:45:52.845 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:45:53.002 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1964430117,v1:127.0.0.1:6801/1964430117] [v2:127.0.0.1:6802/1964430117,v1:127.0.0.1:6803/1964430117] exists,up 2331e813-4a0e-403c-96d2-3ed8bc19bc4c 2026-02-13T02:45:53.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:45:53.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:45:53.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:45:53.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:425: TEST_scrub_permit_time: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:45:53.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:427: TEST_scrub_permit_time: run_osd td/osd-scrub-test 1 --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=0 --osd_scrub_end_hour=1 2026-02-13T02:45:53.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-13T02:45:53.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:45:53.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-13T02:45:53.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:45:53.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/1 2026-02-13T02:45:53.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-13T02:45:53.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:45:53.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:45:53.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:45:53.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2026-02-13T02:45:53.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2026-02-13T02:45:53.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:45:53.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:45:53.003 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-13T02:45:53.003 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:45:53.003 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:45:53.003 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:45:53.004 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:45:53.004 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:45:53.004 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:45:53.004 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:45:53.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:45:53.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:45:53.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:45:53.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:45:53.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-13T02:45:53.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-13T02:45:53.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:45:53.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:45:53.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:45:53.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:45:53.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:45:53.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=0 --osd_scrub_end_hour=1' 2026-02-13T02:45:53.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/1 2026-02-13T02:45:53.006 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:45:53.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=08cdd4fc-9864-4b57-ad69-9257fddaa56d 2026-02-13T02:45:53.007 INFO:tasks.workunit.client.0.trial112.stdout:add osd1 08cdd4fc-9864-4b57-ad69-9257fddaa56d 2026-02-13T02:45:53.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 08cdd4fc-9864-4b57-ad69-9257fddaa56d' 2026-02-13T02:45:53.007 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:45:53.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBhkI5psMX2ABAAYAdVDTUHUjg3ZSYHblTXrQ== 2026-02-13T02:45:53.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBhkI5psMX2ABAAYAdVDTUHUjg3ZSYHblTXrQ=="}' 2026-02-13T02:45:53.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 08cdd4fc-9864-4b57-ad69-9257fddaa56d -i td/osd-scrub-test/1/new.json 2026-02-13T02:45:53.184 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:45:53.196 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/1/new.json 2026-02-13T02:45:53.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=0 --osd_scrub_end_hour=1 --mkfs --key AQBhkI5psMX2ABAAYAdVDTUHUjg3ZSYHblTXrQ== --osd-uuid 08cdd4fc-9864-4b57-ad69-9257fddaa56d 2026-02-13T02:45:53.221 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:45:53.216+0000 7f1992bd5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:45:53.223 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:45:53.216+0000 7f1992bd5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:45:53.224 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:45:53.216+0000 7f1992bd5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:45:53.224 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:45:53.216+0000 7f1992bd5a40 -1 bdev(0x55cf77bf8c00 td/osd-scrub-test/1/block) open stat got: (1) Operation not permitted 2026-02-13T02:45:53.224 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:45:53.216+0000 7f1992bd5a40 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2026-02-13T02:45:53.872 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2026-02-13T02:45:53.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:45:53.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-13T02:45:53.874 INFO:tasks.workunit.client.0.trial112.stdout:adding osd1 key to auth repository 2026-02-13T02:45:53.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:45:54.072 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-13T02:45:54.072 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T02:45:54.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=0 --osd_scrub_end_hour=1 2026-02-13T02:45:54.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:45:54.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:45:54.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:45:54.088 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:45:54.080+0000 7f19c4efca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:45:54.089 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:45:54.084+0000 7f19c4efca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:45:54.090 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:45:54.084+0000 7f19c4efca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:45:54.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-13T02:45:54.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:45:54.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:45:54.234 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:45:54.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:45:54.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:45:54.235 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:45:54.235 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:45:54.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:45:54.236 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:45:54.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:45:54.519 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:45:54.512+0000 7f19c4efca40 -1 Falling back to public interface 2026-02-13T02:45:55.115 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:45:55.108+0000 7f19c4efca40 -1 osd.1 0 log_to_monitors true 2026-02-13T02:45:55.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:45:55.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:45:55.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:45:55.393 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:45:55.393 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:45:55.393 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:45:55.556 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:45:56.558 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:45:56.558 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:45:56.558 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:45:56.559 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:45:56.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:45:56.559 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:45:56.718 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/785818857,v1:127.0.0.1:6811/785818857] [v2:127.0.0.1:6812/785818857,v1:127.0.0.1:6813/785818857] exists,up 08cdd4fc-9864-4b57-ad69-9257fddaa56d 2026-02-13T02:45:56.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:45:56.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:45:56.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:45:56.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:425: TEST_scrub_permit_time: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:45:56.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:427: TEST_scrub_permit_time: run_osd td/osd-scrub-test 2 --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=0 --osd_scrub_end_hour=1 2026-02-13T02:45:56.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-13T02:45:56.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:45:56.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-13T02:45:56.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:45:56.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/2 2026-02-13T02:45:56.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-13T02:45:56.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:45:56.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:45:56.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:45:56.723 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2026-02-13T02:45:56.723 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2026-02-13T02:45:56.723 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:45:56.723 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:45:56.723 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-13T02:45:56.723 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:45:56.723 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:45:56.723 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:45:56.723 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:45:56.723 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:45:56.723 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:45:56.723 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:45:56.723 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:45:56.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:45:56.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:45:56.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:45:56.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-13T02:45:56.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-13T02:45:56.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:45:56.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:45:56.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:45:56.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:45:56.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:45:56.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=0 --osd_scrub_end_hour=1' 2026-02-13T02:45:56.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/2 2026-02-13T02:45:56.724 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:45:56.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8d2ae49e-eb50-48a6-8f8e-dc469c117fc7 2026-02-13T02:45:56.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 8d2ae49e-eb50-48a6-8f8e-dc469c117fc7' 2026-02-13T02:45:56.726 INFO:tasks.workunit.client.0.trial112.stdout:add osd2 8d2ae49e-eb50-48a6-8f8e-dc469c117fc7 2026-02-13T02:45:56.726 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:45:56.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBkkI5piTmHKxAA0UdV+nwOJO9Gk+lUE5UY7w== 2026-02-13T02:45:56.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBkkI5piTmHKxAA0UdV+nwOJO9Gk+lUE5UY7w=="}' 2026-02-13T02:45:56.734 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8d2ae49e-eb50-48a6-8f8e-dc469c117fc7 -i td/osd-scrub-test/2/new.json 2026-02-13T02:45:56.890 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:45:56.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/2/new.json 2026-02-13T02:45:56.901 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=0 --osd_scrub_end_hour=1 --mkfs --key AQBkkI5piTmHKxAA0UdV+nwOJO9Gk+lUE5UY7w== --osd-uuid 8d2ae49e-eb50-48a6-8f8e-dc469c117fc7 2026-02-13T02:45:56.917 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:45:56.908+0000 7fa5f8492a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:45:56.918 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:45:56.912+0000 7fa5f8492a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:45:56.920 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:45:56.912+0000 7fa5f8492a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:45:56.920 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:45:56.912+0000 7fa5f8492a40 -1 bdev(0x556b6513cc00 td/osd-scrub-test/2/block) open stat got: (1) Operation not permitted 2026-02-13T02:45:56.920 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:45:56.912+0000 7fa5f8492a40 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2026-02-13T02:45:57.523 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2026-02-13T02:45:57.523 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:45:57.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-13T02:45:57.524 INFO:tasks.workunit.client.0.trial112.stdout:adding osd2 key to auth repository 2026-02-13T02:45:57.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:45:57.731 INFO:tasks.workunit.client.0.trial112.stdout:start osd.2 2026-02-13T02:45:57.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-13T02:45:57.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=0 --osd_scrub_end_hour=1 2026-02-13T02:45:57.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:45:57.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:45:57.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:45:57.755 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:45:57.748+0000 7f314c359a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:45:57.756 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:45:57.748+0000 7f314c359a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:45:57.757 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:45:57.752+0000 7f314c359a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:45:57.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-13T02:45:57.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:45:57.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T02:45:57.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:45:57.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:45:57.890 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:45:57.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:45:57.890 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:45:57.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:45:57.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:45:58.053 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:45:58.143 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:45:58.136+0000 7f314c359a40 -1 Falling back to public interface 2026-02-13T02:45:58.729 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:45:58.720+0000 7f314c359a40 -1 osd.2 0 log_to_monitors true 2026-02-13T02:45:59.054 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:45:59.054 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:45:59.055 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:45:59.055 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:45:59.055 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:45:59.055 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:45:59.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:46:00.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:46:00.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:46:00.217 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:46:00.218 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:46:00.218 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:46:00.218 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:46:00.377 INFO:tasks.workunit.client.0.trial112.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/467042542,v1:127.0.0.1:6819/467042542] [v2:127.0.0.1:6820/467042542,v1:127.0.0.1:6821/467042542] exists,up 8d2ae49e-eb50-48a6-8f8e-dc469c117fc7 2026-02-13T02:46:00.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:46:00.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:46:00.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:46:00.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:435: TEST_scrub_permit_time: create_pool test 1 1 2026-02-13T02:46:00.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2026-02-13T02:46:00.587 INFO:tasks.workunit.client.0.trial112.stderr:pool 'test' created 2026-02-13T02:46:00.597 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:46:01.598 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: wait_for_clean 2026-02-13T02:46:01.598 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:46:01.598 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:46:01.598 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:46:01.599 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:46:01.600 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:46:01.600 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:46:01.600 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:46:01.601 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:46:01.601 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:46:01.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:46:01.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:46:01.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:46:01.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:46:01.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:46:01.716 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:46:01.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:46:01.882 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:46:01.882 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:46:01.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:46:01.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:46:01.883 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:46:01.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836490 2026-02-13T02:46:01.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836490 2026-02-13T02:46:01.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836490' 2026-02-13T02:46:01.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:46:01.956 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:46:02.027 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672967 2026-02-13T02:46:02.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672967 2026-02-13T02:46:02.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836490 1-42949672967' 2026-02-13T02:46:02.028 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:46:02.028 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:46:02.099 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509444 2026-02-13T02:46:02.099 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509444 2026-02-13T02:46:02.099 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836490 1-42949672967 2-64424509444' 2026-02-13T02:46:02.099 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:46:02.100 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836490 2026-02-13T02:46:02.100 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:46:02.101 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:46:02.102 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836490 2026-02-13T02:46:02.102 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:46:02.103 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836490 2026-02-13T02:46:02.104 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2026-02-13T02:46:02.104 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836490 2026-02-13T02:46:02.104 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:46:02.270 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836490 2026-02-13T02:46:02.271 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:46:03.272 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:46:03.272 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:46:03.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836490 2026-02-13T02:46:03.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:46:04.433 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 499 -eq 0 ']' 2026-02-13T02:46:04.434 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:46:04.601 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836492 -lt 21474836490 2026-02-13T02:46:04.601 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:46:04.602 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672967 2026-02-13T02:46:04.602 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:46:04.603 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:46:04.604 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672967 2026-02-13T02:46:04.604 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:46:04.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672967 2026-02-13T02:46:04.606 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2026-02-13T02:46:04.606 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 42949672967 2026-02-13T02:46:04.606 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:46:04.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672969 -lt 42949672967 2026-02-13T02:46:04.763 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:46:04.764 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509444 2026-02-13T02:46:04.764 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:46:04.765 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:46:04.766 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509444 2026-02-13T02:46:04.766 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:46:04.768 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509444 2026-02-13T02:46:04.768 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2026-02-13T02:46:04.768 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 64424509444 2026-02-13T02:46:04.769 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:46:04.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509445 -lt 64424509444 2026-02-13T02:46:04.941 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:46:04.942 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:46:04.942 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:46:05.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:46:05.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:46:05.138 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:46:05.139 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:46:05.139 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:46:05.139 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:46:05.139 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:46:05.139 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:46:05.301 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:46:05.302 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:46:05.302 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:46:05.302 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:46:05.502 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:46:05.502 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:46:05.502 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:46:05.503 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:439: TEST_scrub_permit_time: get_pg test SOMETHING 2026-02-13T02:46:05.503 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=test 2026-02-13T02:46:05.503 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-13T02:46:05.503 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map test SOMETHING 2026-02-13T02:46:05.503 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-13T02:46:05.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:439: TEST_scrub_permit_time: local pgid=1.0 2026-02-13T02:46:05.669 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:440: TEST_scrub_permit_time: get_primary test SOMETHING 2026-02-13T02:46:05.669 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2026-02-13T02:46:05.669 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-13T02:46:05.669 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T02:46:05.669 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test SOMETHING 2026-02-13T02:46:05.830 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:440: TEST_scrub_permit_time: local primary=1 2026-02-13T02:46:05.831 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:441: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-13T02:46:05.831 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:46:05.831 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:46:05.831 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:46:05.831 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:46:05.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:441: TEST_scrub_permit_time: local last_scrub=2026-02-13T02:46:00.583164+0000 2026-02-13T02:46:05.990 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:446: TEST_scrub_permit_time: ceph tell 1.0 schedule-scrub 86400 2026-02-13T02:46:06.057 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:46:06.057 INFO:tasks.workunit.client.0.trial112.stdout: "deep": false, 2026-02-13T02:46:06.057 INFO:tasks.workunit.client.0.trial112.stdout: "must": false, 2026-02-13T02:46:06.057 INFO:tasks.workunit.client.0.trial112.stdout: "stamp": "2026-02-12T02:46:06.054493+0000" 2026-02-13T02:46:06.058 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:46:06.065 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i=0 )) 2026-02-13T02:46:06.065 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-13T02:46:06.066 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-13T02:46:06.066 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:46:06.066 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:46:06.066 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:46:06.066 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:46:06.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-13T02:46:00.583164+0000 '>' 2026-02-13T02:46:00.583164+0000 2026-02-13T02:46:06.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-13T02:46:07.228 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-13T02:46:07.228 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-13T02:46:07.228 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-13T02:46:07.228 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:46:07.228 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:46:07.229 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:46:07.229 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:46:07.385 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-13T02:46:00.583164+0000 '>' 2026-02-13T02:46:00.583164+0000 2026-02-13T02:46:07.385 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-13T02:46:08.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-13T02:46:08.386 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-13T02:46:08.387 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-13T02:46:08.387 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:46:08.387 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:46:08.388 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:46:08.388 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:46:08.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-12T02:46:06.054493+0000 '>' 2026-02-13T02:46:00.583164+0000 2026-02-13T02:46:08.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-13T02:46:09.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-13T02:46:09.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-13T02:46:09.546 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-13T02:46:09.546 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:46:09.546 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:46:09.547 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:46:09.547 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:46:09.705 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-12T02:46:06.054493+0000 '>' 2026-02-13T02:46:00.583164+0000 2026-02-13T02:46:09.705 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-13T02:46:10.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-13T02:46:10.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-13T02:46:10.707 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-13T02:46:10.707 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:46:10.707 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:46:10.707 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:46:10.707 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:46:10.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-12T02:46:06.054493+0000 '>' 2026-02-13T02:46:00.583164+0000 2026-02-13T02:46:10.862 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-13T02:46:11.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-13T02:46:11.863 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-13T02:46:11.863 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-13T02:46:11.863 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:46:11.863 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:46:11.863 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:46:11.864 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:46:12.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-12T02:46:06.054493+0000 '>' 2026-02-13T02:46:00.583164+0000 2026-02-13T02:46:12.018 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-13T02:46:13.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-13T02:46:13.019 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-13T02:46:13.020 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-13T02:46:13.020 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:46:13.021 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:46:13.021 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:46:13.021 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:46:13.176 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-12T02:46:06.054493+0000 '>' 2026-02-13T02:46:00.583164+0000 2026-02-13T02:46:13.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-13T02:46:14.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-13T02:46:14.177 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-13T02:46:14.178 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-13T02:46:14.178 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:46:14.178 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:46:14.179 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:46:14.179 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:46:14.343 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-12T02:46:06.054493+0000 '>' 2026-02-13T02:46:00.583164+0000 2026-02-13T02:46:14.343 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-13T02:46:15.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-13T02:46:15.344 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-13T02:46:15.344 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-13T02:46:15.344 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:46:15.344 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:46:15.345 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:46:15.345 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:46:15.505 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-12T02:46:06.054493+0000 '>' 2026-02-13T02:46:00.583164+0000 2026-02-13T02:46:15.505 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-13T02:46:16.506 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-13T02:46:16.507 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-13T02:46:16.508 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-13T02:46:16.508 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:46:16.508 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:46:16.508 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:46:16.508 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:46:16.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-12T02:46:06.054493+0000 '>' 2026-02-13T02:46:00.583164+0000 2026-02-13T02:46:16.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-13T02:46:17.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-13T02:46:17.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-13T02:46:17.666 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-13T02:46:17.666 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:46:17.666 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:46:17.667 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:46:17.667 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:46:17.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-12T02:46:06.054493+0000 '>' 2026-02-13T02:46:00.583164+0000 2026-02-13T02:46:17.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-13T02:46:18.823 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-13T02:46:18.823 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-13T02:46:18.824 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-13T02:46:18.824 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:46:18.824 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:46:18.824 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:46:18.824 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:46:18.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-12T02:46:06.054493+0000 '>' 2026-02-13T02:46:00.583164+0000 2026-02-13T02:46:18.979 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-13T02:46:19.980 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-13T02:46:19.980 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-13T02:46:19.980 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-13T02:46:19.980 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:46:19.980 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:46:19.981 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:46:19.981 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:46:20.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-12T02:46:06.054493+0000 '>' 2026-02-13T02:46:00.583164+0000 2026-02-13T02:46:20.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-13T02:46:21.138 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-13T02:46:21.138 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-13T02:46:21.138 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-13T02:46:21.139 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:46:21.139 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:46:21.139 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:46:21.139 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:46:21.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-12T02:46:06.054493+0000 '>' 2026-02-13T02:46:00.583164+0000 2026-02-13T02:46:21.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-13T02:46:22.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-13T02:46:22.295 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-13T02:46:22.296 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-13T02:46:22.296 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:46:22.296 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:46:22.296 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:46:22.296 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:46:22.452 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-12T02:46:06.054493+0000 '>' 2026-02-13T02:46:00.583164+0000 2026-02-13T02:46:22.453 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-13T02:46:23.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-13T02:46:23.454 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-13T02:46:23.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-13T02:46:23.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:46:23.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:46:23.455 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:46:23.456 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:46:23.611 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-12T02:46:06.054493+0000 '>' 2026-02-13T02:46:00.583164+0000 2026-02-13T02:46:23.611 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-13T02:46:24.613 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-13T02:46:24.613 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-13T02:46:24.614 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-13T02:46:24.614 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:46:24.614 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:46:24.614 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:46:24.614 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:46:24.774 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-12T02:46:06.054493+0000 '>' 2026-02-13T02:46:00.583164+0000 2026-02-13T02:46:24.774 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-13T02:46:25.775 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-13T02:46:25.775 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-13T02:46:25.776 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-13T02:46:25.776 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:46:25.776 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:46:25.776 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:46:25.776 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:46:25.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-12T02:46:06.054493+0000 '>' 2026-02-13T02:46:00.583164+0000 2026-02-13T02:46:25.939 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-13T02:46:26.940 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-13T02:46:26.941 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-13T02:46:26.942 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-13T02:46:26.942 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:46:26.942 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:46:26.942 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:46:26.942 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:46:27.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-12T02:46:06.054493+0000 '>' 2026-02-13T02:46:00.583164+0000 2026-02-13T02:46:27.102 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-13T02:46:28.104 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-13T02:46:28.104 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-13T02:46:28.105 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-13T02:46:28.105 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:46:28.105 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:46:28.105 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:46:28.105 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:46:28.267 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-12T02:46:06.054493+0000 '>' 2026-02-13T02:46:00.583164+0000 2026-02-13T02:46:28.267 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-13T02:46:29.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-13T02:46:29.268 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-13T02:46:29.269 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-13T02:46:29.269 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:46:29.269 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:46:29.269 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:46:29.269 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:46:29.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-12T02:46:06.054493+0000 '>' 2026-02-13T02:46:00.583164+0000 2026-02-13T02:46:29.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-13T02:46:30.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-13T02:46:30.434 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-13T02:46:30.435 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-13T02:46:30.435 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:46:30.435 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:46:30.435 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:46:30.435 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:46:30.596 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-12T02:46:06.054493+0000 '>' 2026-02-13T02:46:00.583164+0000 2026-02-13T02:46:30.596 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-13T02:46:31.597 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-13T02:46:31.597 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-13T02:46:31.597 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-13T02:46:31.598 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:46:31.598 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:46:31.598 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:46:31.598 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:46:31.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-12T02:46:06.054493+0000 '>' 2026-02-13T02:46:00.583164+0000 2026-02-13T02:46:31.759 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-13T02:46:32.760 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-13T02:46:32.761 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-13T02:46:32.761 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-13T02:46:32.761 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:46:32.762 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:46:32.762 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:46:32.762 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:46:32.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-12T02:46:06.054493+0000 '>' 2026-02-13T02:46:00.583164+0000 2026-02-13T02:46:32.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-13T02:46:33.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-13T02:46:33.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-13T02:46:33.923 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-13T02:46:33.923 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:46:33.923 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:46:33.923 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:46:33.923 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:46:34.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-12T02:46:06.054493+0000 '>' 2026-02-13T02:46:00.583164+0000 2026-02-13T02:46:34.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-13T02:46:35.086 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-13T02:46:35.086 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-13T02:46:35.087 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-13T02:46:35.087 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:46:35.087 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:46:35.087 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:46:35.087 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:46:35.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-12T02:46:06.054493+0000 '>' 2026-02-13T02:46:00.583164+0000 2026-02-13T02:46:35.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-13T02:46:36.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-13T02:46:36.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-13T02:46:36.246 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-13T02:46:36.246 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:46:36.246 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:46:36.247 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:46:36.247 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:46:36.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-12T02:46:06.054493+0000 '>' 2026-02-13T02:46:00.583164+0000 2026-02-13T02:46:36.404 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-13T02:46:37.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-13T02:46:37.405 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-13T02:46:37.406 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-13T02:46:37.406 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:46:37.406 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:46:37.406 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:46:37.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:46:37.561 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-12T02:46:06.054493+0000 '>' 2026-02-13T02:46:00.583164+0000 2026-02-13T02:46:37.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-13T02:46:38.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-13T02:46:38.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-13T02:46:38.564 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-13T02:46:38.564 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:46:38.564 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:46:38.564 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:46:38.564 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:46:38.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-12T02:46:06.054493+0000 '>' 2026-02-13T02:46:00.583164+0000 2026-02-13T02:46:38.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-13T02:46:39.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-13T02:46:39.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-13T02:46:39.726 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-13T02:46:39.726 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:46:39.726 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:46:39.726 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:46:39.726 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:46:39.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-12T02:46:06.054493+0000 '>' 2026-02-13T02:46:00.583164+0000 2026-02-13T02:46:39.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-13T02:46:40.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-13T02:46:40.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-13T02:46:40.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:455: TEST_scrub_permit_time: perf_counters td/osd-scrub-test 3 2026-02-13T02:46:40.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2026-02-13T02:46:40.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=3 2026-02-13T02:46:40.892 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 3 - 1 2026-02-13T02:46:40.893 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 2 2026-02-13T02:46:40.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:46:40.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2026-02-13T02:46:40.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-13T02:46:40.967 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:46:40.967 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_ec": [ 2026-02-13T02:46:40.967 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:46:40.967 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:46:40.967 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:46:40.968 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:46:40.968 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:46:40.968 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:46:40.968 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:46:40.968 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:46:40.968 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:46:40.968 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:46:40.968 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:46:40.968 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:46:40.968 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:46:40.968 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:46:40.968 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_repl": [ 2026-02-13T02:46:40.968 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:46:40.969 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:46:40.969 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:46:40.969 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:46:40.969 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:46:40.969 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:46:40.969 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:46:40.969 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:46:40.969 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:46:40.969 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:46:40.969 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:46:40.969 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:46:40.969 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:46:40.970 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:46:40.970 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_ec": [ 2026-02-13T02:46:40.970 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:46:40.970 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:46:40.970 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:46:40.970 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:46:40.970 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:46:40.970 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:46:40.970 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:46:40.970 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:46:40.970 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:46:40.970 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:46:40.971 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:46:40.971 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:46:40.971 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:46:40.971 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:46:40.971 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_repl": [ 2026-02-13T02:46:40.971 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:46:40.971 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:46:40.971 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:46:40.971 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:46:40.971 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:46:40.971 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:46:40.971 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:46:40.971 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:46:40.971 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:46:40.972 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:46:40.972 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:46:40.972 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:46:40.972 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:46:40.972 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:46:40.972 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:46:40.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:46:40.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2026-02-13T02:46:40.973 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-13T02:46:41.039 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:46:41.039 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_ec": [ 2026-02-13T02:46:41.039 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:46:41.040 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:46:41.040 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:46:41.040 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:46:41.040 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:46:41.040 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:46:41.040 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:46:41.040 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:46:41.040 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:46:41.040 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:46:41.040 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:46:41.040 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:46:41.040 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:46:41.040 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:46:41.041 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_repl": [ 2026-02-13T02:46:41.041 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:46:41.041 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:46:41.041 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:46:41.041 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:46:41.041 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:46:41.041 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:46:41.041 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:46:41.041 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:46:41.041 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:46:41.041 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:46:41.041 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:46:41.042 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:46:41.042 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:46:41.042 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:46:41.042 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_ec": [ 2026-02-13T02:46:41.042 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:46:41.042 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:46:41.042 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:46:41.042 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:46:41.042 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:46:41.042 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:46:41.042 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:46:41.042 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:46:41.042 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:46:41.042 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:46:41.043 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:46:41.043 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:46:41.043 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:46:41.043 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:46:41.043 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_repl": [ 2026-02-13T02:46:41.043 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:46:41.043 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:46:41.043 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:46:41.043 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:46:41.043 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:46:41.043 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:46:41.043 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:46:41.043 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:46:41.044 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:46:41.044 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:46:41.044 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:46:41.044 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:46:41.044 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:46:41.044 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:46:41.044 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:46:41.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:46:41.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.2 counter dump 2026-02-13T02:46:41.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-13T02:46:41.112 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:46:41.112 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_ec": [ 2026-02-13T02:46:41.112 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:46:41.113 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:46:41.113 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:46:41.113 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:46:41.113 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:46:41.113 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:46:41.113 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:46:41.113 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:46:41.113 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:46:41.113 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:46:41.113 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:46:41.113 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:46:41.113 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:46:41.113 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:46:41.114 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_repl": [ 2026-02-13T02:46:41.114 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:46:41.114 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:46:41.114 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:46:41.114 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:46:41.114 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:46:41.114 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:46:41.114 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:46:41.114 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:46:41.114 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:46:41.114 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:46:41.114 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:46:41.115 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:46:41.115 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:46:41.115 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:46:41.115 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_ec": [ 2026-02-13T02:46:41.115 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:46:41.115 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:46:41.115 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:46:41.115 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:46:41.115 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:46:41.115 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:46:41.115 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:46:41.115 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:46:41.115 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:46:41.116 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:46:41.116 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:46:41.116 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:46:41.116 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:46:41.116 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:46:41.116 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_repl": [ 2026-02-13T02:46:41.116 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:46:41.116 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:46:41.116 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:46:41.116 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:46:41.116 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:46:41.116 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:46:41.116 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:46:41.117 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:46:41.117 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:46:41.117 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:46:41.117 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:46:41.117 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:46:41.117 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:46:41.117 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:46:41.117 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:46:41.117 INFO:tasks.workunit.client.0.trial112.stdout:-------------- Teardown Test TEST_scrub_permit_time ------------------ 2026-02-13T02:46:41.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_scrub_permit_time ------------------' 2026-02-13T02:46:41.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2026-02-13T02:46:41.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-13T02:46:41.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:46:41.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-13T02:46:41.119 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:46:41.119 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:46:41.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:46:41.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:46:41.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:46:41.223 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:46:41.223 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:46:41.224 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:46:41.224 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:46:41.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:46:41.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:46:41.226 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:46:41.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:46:41.227 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:46:41.228 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:46:41.228 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:46:41.229 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:46:41.230 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:46:41.230 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-13T02:46:41.239 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:46:41.239 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:46:41.240 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:46:41.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.508946 2026-02-13T02:46:41.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:46:41.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:46:41.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_scrub_permit_time ------------------' 2026-02-13T02:46:41.242 INFO:tasks.workunit.client.0.trial112.stdout:-------------- Complete Test TEST_scrub_permit_time ------------------ 2026-02-13T02:46:41.242 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2026-02-13T02:46:41.242 INFO:tasks.workunit.client.0.trial112.stdout:-------------- Prepare Test TEST_scrub_test ------------------- 2026-02-13T02:46:41.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_scrub_test -------------------' 2026-02-13T02:46:41.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2026-02-13T02:46:41.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2026-02-13T02:46:41.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2026-02-13T02:46:41.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-13T02:46:41.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:46:41.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-13T02:46:41.244 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:46:41.244 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:46:41.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:46:41.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:46:41.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:46:41.245 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:46:41.245 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:46:41.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:46:41.246 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:46:41.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:46:41.248 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:46:41.248 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:46:41.249 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:46:41.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:46:41.250 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:46:41.250 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:46:41.251 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:46:41.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:46:41.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-13T02:46:41.254 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:46:41.254 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:46:41.254 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:46:41.254 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.508946 2026-02-13T02:46:41.255 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:46:41.255 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:46:41.255 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2026-02-13T02:46:41.257 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T02:46:41.257 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:46:41.257 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:46:41.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.508946 2026-02-13T02:46:41.258 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T02:46:41.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-13T02:46:41.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T02:46:41.259 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2026-02-13T02:46:41.260 INFO:tasks.workunit.client.0.trial112.stdout:-------------- Run Test TEST_scrub_test ----------------------- 2026-02-13T02:46:41.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_scrub_test -----------------------' 2026-02-13T02:46:41.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_scrub_test td/osd-scrub-test 2026-02-13T02:46:41.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:52: TEST_scrub_test: local dir=td/osd-scrub-test 2026-02-13T02:46:41.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:53: TEST_scrub_test: local poolname=test 2026-02-13T02:46:41.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:54: TEST_scrub_test: local OSDS=3 2026-02-13T02:46:41.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:55: TEST_scrub_test: local objects=15 2026-02-13T02:46:41.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:57: TEST_scrub_test: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2026-02-13T02:46:41.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-test 2026-02-13T02:46:41.261 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-13T02:46:41.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-13T02:46:41.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-13T02:46:41.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-test/a 2026-02-13T02:46:41.262 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2026-02-13T02:46:41.280 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-13T02:46:41.280 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:46:41.280 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:46:41.281 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:46:41.281 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:46:41.282 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:46:41.282 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:46:41.282 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2026-02-13T02:46:41.308 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-13T02:46:41.309 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-13T02:46:41.309 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:46:41.309 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:46:41.309 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-13T02:46:41.311 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-13T02:46:41.311 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:46:41.311 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:46:41.311 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:46:41.311 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:46:41.311 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:46:41.311 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:46:41.311 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.508946/ceph-mon.a.asok 2026-02-13T02:46:41.312 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:46:41.312 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.508946/ceph-mon.a.asok config get fsid 2026-02-13T02:46:41.367 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-13T02:46:41.367 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:46:41.367 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:46:41.367 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-13T02:46:41.367 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-13T02:46:41.369 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:46:41.369 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:46:41.369 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:46:41.369 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:46:41.369 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:46:41.369 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:46:41.369 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.508946/ceph-mon.a.asok 2026-02-13T02:46:41.369 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:46:41.369 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.508946/ceph-mon.a.asok config get mon_host 2026-02-13T02:46:41.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:58: TEST_scrub_test: run_mgr td/osd-scrub-test x --mgr_stats_period=1 2026-02-13T02:46:41.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-test 2026-02-13T02:46:41.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-13T02:46:41.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-13T02:46:41.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-13T02:46:41.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-test/x 2026-02-13T02:46:41.423 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-13T02:46:41.521 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-13T02:46:41.521 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:46:41.521 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:46:41.521 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:46:41.521 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:46:41.521 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:46:41.522 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:46:41.522 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:46:41.523 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mgr_stats_period=1 2026-02-13T02:46:41.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:59: TEST_scrub_test: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 ' 2026-02-13T02:46:41.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:60: TEST_scrub_test: ceph_osd_args+='--osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 ' 2026-02-13T02:46:41.540 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:61: TEST_scrub_test: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2026-02-13T02:46:41.541 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:62: TEST_scrub_test: expr 3 - 1 2026-02-13T02:46:41.541 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:62: TEST_scrub_test: seq 0 2 2026-02-13T02:46:41.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:62: TEST_scrub_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:46:41.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:64: TEST_scrub_test: run_osd td/osd-scrub-test 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2026-02-13T02:46:41.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-13T02:46:41.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:46:41.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-13T02:46:41.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:46:41.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/0 2026-02-13T02:46:41.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-13T02:46:41.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:46:41.545 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:46:41.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:46:41.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2026-02-13T02:46:41.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2026-02-13T02:46:41.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:46:41.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:46:41.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-13T02:46:41.546 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:46:41.546 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:46:41.546 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:46:41.546 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:46:41.546 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:46:41.546 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:46:41.546 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:46:41.547 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:46:41.547 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:46:41.547 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:46:41.547 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:46:41.547 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-13T02:46:41.547 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-13T02:46:41.547 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:46:41.547 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:46:41.547 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:46:41.547 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:46:41.547 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:46:41.547 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2026-02-13T02:46:41.547 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/0 2026-02-13T02:46:41.548 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:46:41.548 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=04faf3a3-9b36-405c-9ac3-49e018150fc5 2026-02-13T02:46:41.548 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 04faf3a3-9b36-405c-9ac3-49e018150fc5' 2026-02-13T02:46:41.549 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 04faf3a3-9b36-405c-9ac3-49e018150fc5 2026-02-13T02:46:41.549 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:46:41.557 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCRkI5pvdsHIRAAzA9U5idOQ4SMipzxn7Pkbg== 2026-02-13T02:46:41.558 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCRkI5pvdsHIRAAzA9U5idOQ4SMipzxn7Pkbg=="}' 2026-02-13T02:46:41.558 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 04faf3a3-9b36-405c-9ac3-49e018150fc5 -i td/osd-scrub-test/0/new.json 2026-02-13T02:46:41.649 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:46:41.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/0/new.json 2026-02-13T02:46:41.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCRkI5pvdsHIRAAzA9U5idOQ4SMipzxn7Pkbg== --osd-uuid 04faf3a3-9b36-405c-9ac3-49e018150fc5 2026-02-13T02:46:41.675 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:46:41.667+0000 7f6817858a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:46:41.676 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:46:41.667+0000 7f6817858a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:46:41.677 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:46:41.671+0000 7f6817858a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:46:41.677 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:46:41.671+0000 7f6817858a40 -1 bdev(0x562af1474c00 td/osd-scrub-test/0/block) open stat got: (1) Operation not permitted 2026-02-13T02:46:41.678 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:46:41.671+0000 7f6817858a40 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2026-02-13T02:46:42.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2026-02-13T02:46:42.437 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:46:42.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-13T02:46:42.438 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T02:46:42.438 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:46:42.541 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T02:46:42.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-13T02:46:42.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2026-02-13T02:46:42.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:46:42.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:46:42.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:46:42.567 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:46:42.559+0000 7f2dd85b4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:46:42.569 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:46:42.563+0000 7f2dd85b4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:46:42.570 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:46:42.563+0000 7f2dd85b4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:46:42.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-13T02:46:42.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:46:42.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:46:42.632 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:46:42.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:46:42.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:46:42.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:46:42.633 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:46:42.633 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:46:42.634 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:46:42.722 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:46:42.967 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:46:42.959+0000 7f2dd85b4a40 -1 Falling back to public interface 2026-02-13T02:46:43.723 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:46:43.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:46:43.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:46:43.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:46:43.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:46:43.725 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:46:43.823 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:46:43.839 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:46:43.831+0000 7f2dd85b4a40 -1 osd.0 0 log_to_monitors true 2026-02-13T02:46:44.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:46:44.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:46:44.824 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:46:44.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:46:44.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:46:44.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:46:44.980 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/4197162300,v1:127.0.0.1:6801/4197162300] [v2:127.0.0.1:6802/4197162300,v1:127.0.0.1:6803/4197162300] exists,up 04faf3a3-9b36-405c-9ac3-49e018150fc5 2026-02-13T02:46:44.981 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:46:44.981 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:46:44.981 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:46:44.981 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:62: TEST_scrub_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:46:44.981 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:64: TEST_scrub_test: run_osd td/osd-scrub-test 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2026-02-13T02:46:44.981 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-13T02:46:44.981 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:46:44.981 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-13T02:46:44.981 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:46:44.981 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/1 2026-02-13T02:46:44.981 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-13T02:46:44.981 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:46:44.981 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:46:44.981 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:46:44.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2026-02-13T02:46:44.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2026-02-13T02:46:44.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:46:44.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:46:44.982 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-13T02:46:44.982 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:46:44.982 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:46:44.982 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:46:44.982 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:46:44.982 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:46:44.982 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:46:44.983 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:46:44.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:46:44.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:46:44.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:46:44.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:46:44.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-13T02:46:44.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-13T02:46:44.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:46:44.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:46:44.983 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:46:44.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:46:44.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:46:44.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2026-02-13T02:46:44.984 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/1 2026-02-13T02:46:44.985 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:46:44.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=52ee40ce-b657-4d1e-adf7-64bbbd7addfd 2026-02-13T02:46:44.986 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 52ee40ce-b657-4d1e-adf7-64bbbd7addfd' 2026-02-13T02:46:44.986 INFO:tasks.workunit.client.0.trial112.stdout:add osd1 52ee40ce-b657-4d1e-adf7-64bbbd7addfd 2026-02-13T02:46:44.986 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:46:44.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCUkI5p3DZYOxAA4KvnKN9YfglJhVoFLaJpFQ== 2026-02-13T02:46:44.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCUkI5p3DZYOxAA4KvnKN9YfglJhVoFLaJpFQ=="}' 2026-02-13T02:46:44.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 52ee40ce-b657-4d1e-adf7-64bbbd7addfd -i td/osd-scrub-test/1/new.json 2026-02-13T02:46:45.159 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:46:45.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/1/new.json 2026-02-13T02:46:45.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCUkI5p3DZYOxAA4KvnKN9YfglJhVoFLaJpFQ== --osd-uuid 52ee40ce-b657-4d1e-adf7-64bbbd7addfd 2026-02-13T02:46:45.186 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:46:45.179+0000 7f8248590a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:46:45.187 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:46:45.179+0000 7f8248590a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:46:45.188 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:46:45.179+0000 7f8248590a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:46:45.189 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:46:45.179+0000 7f8248590a40 -1 bdev(0x563ae1d30c00 td/osd-scrub-test/1/block) open stat got: (1) Operation not permitted 2026-02-13T02:46:45.189 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:46:45.179+0000 7f8248590a40 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2026-02-13T02:46:45.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2026-02-13T02:46:45.800 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:46:45.801 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-13T02:46:45.802 INFO:tasks.workunit.client.0.trial112.stdout:adding osd1 key to auth repository 2026-02-13T02:46:45.802 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:46:46.005 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-13T02:46:46.006 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T02:46:46.006 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:46:46.006 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2026-02-13T02:46:46.006 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:46:46.007 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:46:46.022 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:46:46.015+0000 7f0f9673ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:46:46.024 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:46:46.015+0000 7f0f9673ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:46:46.025 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:46:46.015+0000 7f0f9673ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:46:46.167 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-13T02:46:46.167 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:46:46.167 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:46:46.167 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:46:46.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:46:46.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:46:46.168 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:46:46.168 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:46:46.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:46:46.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:46:46.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:46:46.379 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:46:46.371+0000 7f0f9673ca40 -1 Falling back to public interface 2026-02-13T02:46:46.973 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:46:46.963+0000 7f0f9673ca40 -1 osd.1 0 log_to_monitors true 2026-02-13T02:46:47.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:46:47.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:46:47.331 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:46:47.331 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:46:47.332 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:46:47.332 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:46:47.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:46:48.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:46:48.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:46:48.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:46:48.492 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:46:48.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:46:48.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:46:48.649 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1146841921,v1:127.0.0.1:6811/1146841921] [v2:127.0.0.1:6812/1146841921,v1:127.0.0.1:6813/1146841921] exists,up 52ee40ce-b657-4d1e-adf7-64bbbd7addfd 2026-02-13T02:46:48.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:46:48.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:46:48.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:46:48.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:62: TEST_scrub_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:46:48.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:64: TEST_scrub_test: run_osd td/osd-scrub-test 2 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2026-02-13T02:46:48.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-13T02:46:48.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:46:48.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-13T02:46:48.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:46:48.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/2 2026-02-13T02:46:48.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-13T02:46:48.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:46:48.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:46:48.653 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:46:48.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2026-02-13T02:46:48.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2026-02-13T02:46:48.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:46:48.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:46:48.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-13T02:46:48.654 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:46:48.654 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:46:48.654 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:46:48.654 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:46:48.654 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:46:48.654 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:46:48.654 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:46:48.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:46:48.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:46:48.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:46:48.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:46:48.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-13T02:46:48.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-13T02:46:48.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:46:48.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:46:48.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:46:48.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:46:48.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:46:48.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2026-02-13T02:46:48.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/2 2026-02-13T02:46:48.656 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:46:48.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=9e0a9018-0d7e-46cb-9a96-1a17a91de220 2026-02-13T02:46:48.657 INFO:tasks.workunit.client.0.trial112.stdout:add osd2 9e0a9018-0d7e-46cb-9a96-1a17a91de220 2026-02-13T02:46:48.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 9e0a9018-0d7e-46cb-9a96-1a17a91de220' 2026-02-13T02:46:48.657 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:46:48.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCYkI5p/Kp7JxAAeYB6h6tNZDehV5PSSPpaJQ== 2026-02-13T02:46:48.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCYkI5p/Kp7JxAAeYB6h6tNZDehV5PSSPpaJQ=="}' 2026-02-13T02:46:48.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 9e0a9018-0d7e-46cb-9a96-1a17a91de220 -i td/osd-scrub-test/2/new.json 2026-02-13T02:46:48.831 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:46:48.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/2/new.json 2026-02-13T02:46:48.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCYkI5p/Kp7JxAAeYB6h6tNZDehV5PSSPpaJQ== --osd-uuid 9e0a9018-0d7e-46cb-9a96-1a17a91de220 2026-02-13T02:46:48.867 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:46:48.859+0000 7f55f467aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:46:48.868 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:46:48.859+0000 7f55f467aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:46:48.869 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:46:48.863+0000 7f55f467aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:46:48.870 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:46:48.863+0000 7f55f467aa40 -1 bdev(0x55700bbf2c00 td/osd-scrub-test/2/block) open stat got: (1) Operation not permitted 2026-02-13T02:46:48.870 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:46:48.863+0000 7f55f467aa40 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2026-02-13T02:46:49.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2026-02-13T02:46:49.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:46:49.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-13T02:46:49.526 INFO:tasks.workunit.client.0.trial112.stdout:adding osd2 key to auth repository 2026-02-13T02:46:49.527 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:46:49.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-13T02:46:49.731 INFO:tasks.workunit.client.0.trial112.stdout:start osd.2 2026-02-13T02:46:49.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2026-02-13T02:46:49.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:46:49.732 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:46:49.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:46:49.748 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:46:49.739+0000 7fe1beff2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:46:49.749 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:46:49.743+0000 7fe1beff2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:46:49.750 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:46:49.743+0000 7fe1beff2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:46:49.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-13T02:46:49.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:46:49.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T02:46:49.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:46:49.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:46:49.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:46:49.895 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:46:49.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:46:49.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:46:49.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:46:50.050 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:46:50.175 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:46:50.167+0000 7fe1beff2a40 -1 Falling back to public interface 2026-02-13T02:46:50.801 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:46:50.791+0000 7fe1beff2a40 -1 osd.2 0 log_to_monitors true 2026-02-13T02:46:51.051 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:46:51.052 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:46:51.052 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:46:51.052 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:46:51.053 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:46:51.053 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:46:51.213 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:46:52.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:46:52.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:46:52.215 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:46:52.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:46:52.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:46:52.216 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:46:52.377 INFO:tasks.workunit.client.0.trial112.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3605577948,v1:127.0.0.1:6819/3605577948] [v2:127.0.0.1:6820/3605577948,v1:127.0.0.1:6821/3605577948] exists,up 9e0a9018-0d7e-46cb-9a96-1a17a91de220 2026-02-13T02:46:52.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:46:52.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:46:52.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:46:52.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:68: TEST_scrub_test: create_pool test 1 1 2026-02-13T02:46:52.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2026-02-13T02:46:52.581 INFO:tasks.workunit.client.0.trial112.stderr:pool 'test' created 2026-02-13T02:46:52.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-13T02:46:53.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:69: TEST_scrub_test: wait_for_clean 2026-02-13T02:46:53.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:46:53.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:46:53.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:46:53.594 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:46:53.595 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:46:53.595 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:46:53.595 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:46:53.595 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:46:53.595 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:46:53.704 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:46:53.704 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:46:53.704 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:46:53.705 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:46:53.705 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:46:53.705 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:46:53.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:46:53.864 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:46:53.864 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:46:53.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:46:53.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:46:53.865 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:46:53.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836490 2026-02-13T02:46:53.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836490 2026-02-13T02:46:53.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836490' 2026-02-13T02:46:53.936 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:46:53.936 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:46:54.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672967 2026-02-13T02:46:54.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672967 2026-02-13T02:46:54.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836490 1-42949672967' 2026-02-13T02:46:54.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:46:54.009 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:46:54.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542148 2026-02-13T02:46:54.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542148 2026-02-13T02:46:54.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836490 1-42949672967 2-60129542148' 2026-02-13T02:46:54.081 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:46:54.082 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836490 2026-02-13T02:46:54.082 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:46:54.083 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:46:54.084 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836490 2026-02-13T02:46:54.084 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:46:54.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836490 2026-02-13T02:46:54.086 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2026-02-13T02:46:54.086 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836490 2026-02-13T02:46:54.086 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:46:54.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836490 2026-02-13T02:46:54.252 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:46:55.254 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:46:55.254 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:46:55.414 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836490 2026-02-13T02:46:55.414 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:46:56.415 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 499 -eq 0 ']' 2026-02-13T02:46:56.416 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:46:56.581 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836492 -lt 21474836490 2026-02-13T02:46:56.581 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:46:56.582 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672967 2026-02-13T02:46:56.582 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:46:56.583 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:46:56.584 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672967 2026-02-13T02:46:56.584 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:46:56.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672967 2026-02-13T02:46:56.586 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2026-02-13T02:46:56.586 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 42949672967 2026-02-13T02:46:56.586 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:46:56.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672969 -lt 42949672967 2026-02-13T02:46:56.751 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:46:56.752 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542148 2026-02-13T02:46:56.752 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:46:56.753 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:46:56.754 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542148 2026-02-13T02:46:56.754 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:46:56.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542148 2026-02-13T02:46:56.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542148' 2026-02-13T02:46:56.756 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 60129542148 2026-02-13T02:46:56.756 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:46:56.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542149 -lt 60129542148 2026-02-13T02:46:56.920 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:46:56.921 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:46:56.921 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:46:57.116 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:46:57.117 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:46:57.117 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:46:57.117 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:46:57.117 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:46:57.117 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:46:57.117 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:46:57.117 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:46:57.277 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:46:57.277 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:46:57.277 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:46:57.277 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:46:57.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:46:57.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:46:57.475 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:46:57.476 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:70: TEST_scrub_test: ceph osd dump 2026-02-13T02:46:57.476 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:70: TEST_scrub_test: grep '^pool.*['\'']test['\'']' 2026-02-13T02:46:57.476 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:70: TEST_scrub_test: awk '{ print $2 }' 2026-02-13T02:46:57.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:70: TEST_scrub_test: poolid=1 2026-02-13T02:46:57.637 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:72: TEST_scrub_test: file_with_random_data 1032 2026-02-13T02:46:57.638 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2344: file_with_random_data: local size_bytes=1032 2026-02-13T02:46:57.638 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2345: file_with_random_data: mktemp 2026-02-13T02:46:57.639 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2345: file_with_random_data: local file=/tmp/tmp.0QUZP0zONP 2026-02-13T02:46:57.639 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2346: file_with_random_data: dd if=/dev/urandom of=/tmp/tmp.0QUZP0zONP bs=1032 count=1 2026-02-13T02:46:57.640 INFO:tasks.workunit.client.0.trial112.stderr:1+0 records in 2026-02-13T02:46:57.641 INFO:tasks.workunit.client.0.trial112.stderr:1+0 records out 2026-02-13T02:46:57.641 INFO:tasks.workunit.client.0.trial112.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 9.0238e-05 s, 11.4 MB/s 2026-02-13T02:46:57.641 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: file_with_random_data: printf %s /tmp/tmp.0QUZP0zONP 2026-02-13T02:46:57.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:72: TEST_scrub_test: local testdata_file=/tmp/tmp.0QUZP0zONP 2026-02-13T02:46:57.641 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: seq 1 15 2026-02-13T02:46:57.642 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-13T02:46:57.642 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj1 /tmp/tmp.0QUZP0zONP 2026-02-13T02:46:57.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-13T02:46:57.662 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj2 /tmp/tmp.0QUZP0zONP 2026-02-13T02:46:57.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-13T02:46:57.685 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj3 /tmp/tmp.0QUZP0zONP 2026-02-13T02:46:57.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-13T02:46:57.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj4 /tmp/tmp.0QUZP0zONP 2026-02-13T02:46:57.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-13T02:46:57.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj5 /tmp/tmp.0QUZP0zONP 2026-02-13T02:46:57.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-13T02:46:57.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj6 /tmp/tmp.0QUZP0zONP 2026-02-13T02:46:57.775 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-13T02:46:57.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj7 /tmp/tmp.0QUZP0zONP 2026-02-13T02:46:57.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-13T02:46:57.798 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj8 /tmp/tmp.0QUZP0zONP 2026-02-13T02:46:57.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-13T02:46:57.822 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj9 /tmp/tmp.0QUZP0zONP 2026-02-13T02:46:57.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-13T02:46:57.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj10 /tmp/tmp.0QUZP0zONP 2026-02-13T02:46:57.870 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-13T02:46:57.870 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj11 /tmp/tmp.0QUZP0zONP 2026-02-13T02:46:57.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-13T02:46:57.895 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj12 /tmp/tmp.0QUZP0zONP 2026-02-13T02:46:57.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-13T02:46:57.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj13 /tmp/tmp.0QUZP0zONP 2026-02-13T02:46:57.943 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-13T02:46:57.944 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj14 /tmp/tmp.0QUZP0zONP 2026-02-13T02:46:57.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-13T02:46:57.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj15 /tmp/tmp.0QUZP0zONP 2026-02-13T02:46:57.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:77: TEST_scrub_test: rm -f /tmp/tmp.0QUZP0zONP 2026-02-13T02:46:57.994 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:79: TEST_scrub_test: get_primary test obj1 2026-02-13T02:46:57.994 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2026-02-13T02:46:57.994 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-13T02:46:57.994 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2026-02-13T02:46:57.995 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T02:46:58.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:79: TEST_scrub_test: local primary=1 2026-02-13T02:46:58.159 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:80: TEST_scrub_test: get_not_primary test obj1 2026-02-13T02:46:58.159 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=test 2026-02-13T02:46:58.159 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=obj1 2026-02-13T02:46:58.159 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary test obj1 2026-02-13T02:46:58.159 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2026-02-13T02:46:58.159 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-13T02:46:58.160 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2026-02-13T02:46:58.160 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T02:46:58.316 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-13T02:46:58.316 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map test obj1 2026-02-13T02:46:58.316 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-13T02:46:58.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:80: TEST_scrub_test: local otherosd=0 2026-02-13T02:46:58.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:81: TEST_scrub_test: '[' 0 = 2 ']' 2026-02-13T02:46:58.476 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:85: TEST_scrub_test: local anotherosd=2 2026-02-13T02:46:58.477 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:88: TEST_scrub_test: file_with_random_data 512 2026-02-13T02:46:58.477 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2344: file_with_random_data: local size_bytes=512 2026-02-13T02:46:58.477 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2345: file_with_random_data: mktemp 2026-02-13T02:46:58.478 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2345: file_with_random_data: local file=/tmp/tmp.6GDRsWdl1y 2026-02-13T02:46:58.478 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2346: file_with_random_data: dd if=/dev/urandom of=/tmp/tmp.6GDRsWdl1y bs=512 count=1 2026-02-13T02:46:58.479 INFO:tasks.workunit.client.0.trial112.stderr:1+0 records in 2026-02-13T02:46:58.479 INFO:tasks.workunit.client.0.trial112.stderr:1+0 records out 2026-02-13T02:46:58.479 INFO:tasks.workunit.client.0.trial112.stderr:512 bytes copied, 8.6281e-05 s, 5.9 MB/s 2026-02-13T02:46:58.481 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: file_with_random_data: printf %s /tmp/tmp.6GDRsWdl1y 2026-02-13T02:46:58.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:88: TEST_scrub_test: local corrupt_data_file=/tmp/tmp.6GDRsWdl1y 2026-02-13T02:46:58.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:89: TEST_scrub_test: objectstore_tool td/osd-scrub-test 2 obj1 set-bytes /tmp/tmp.6GDRsWdl1y 2026-02-13T02:46:58.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-test 2026-02-13T02:46:58.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:46:58.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-13T02:46:58.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:46:58.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-test 2 obj1 set-bytes /tmp/tmp.6GDRsWdl1y 2026-02-13T02:46:58.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-test 2026-02-13T02:46:58.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:46:58.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-13T02:46:58.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:46:58.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-test TERM osd.2 2026-02-13T02:46:58.481 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:46:58.482 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:46:58.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:46:58.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:46:58.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:46:58.789 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:46:58.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-test 2 obj1 set-bytes /tmp/tmp.6GDRsWdl1y 2026-02-13T02:46:58.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-test 2026-02-13T02:46:58.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:46:58.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-13T02:46:58.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:46:58.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-test/2 2026-02-13T02:46:58.790 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-test/2 obj1 set-bytes /tmp/tmp.6GDRsWdl1y 2026-02-13T02:46:59.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-test 2 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2026-02-13T02:46:59.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-test 2026-02-13T02:46:59.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:46:59.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-13T02:46:59.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:46:59.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-test/2 2026-02-13T02:46:59.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-13T02:46:59.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:46:59.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:46:59.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:46:59.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2026-02-13T02:46:59.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2026-02-13T02:46:59.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:46:59.069 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:46:59.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-13T02:46:59.070 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:46:59.070 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:46:59.070 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:46:59.070 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:46:59.070 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:46:59.070 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:46:59.070 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:46:59.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' 2026-02-13T02:46:59.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:46:59.070 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-13T02:46:59.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-13T02:46:59.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:46:59.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:46:59.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:46:59.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:46:59.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:46:59.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2026-02-13T02:46:59.071 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-test/2 2026-02-13T02:46:59.072 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-13T02:46:59.073 INFO:tasks.workunit.client.0.trial112.stderr:start osd.2 2026-02-13T02:46:59.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=7e63d130-1080-401a-a50e-0b1eeacfa653 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.508946/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2026-02-13T02:46:59.073 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-test/2/whoami 2026-02-13T02:46:59.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-13T02:46:59.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:46:59.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:46:59.073 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:46:59.087 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:46:59.079+0000 7fb9dfeeea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:46:59.088 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:46:59.079+0000 7fb9dfeeea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:46:59.090 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:46:59.083+0000 7fb9dfeeea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:46:59.232 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-13T02:46:59.232 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:46:59.232 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T02:46:59.232 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:46:59.232 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:46:59.233 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:46:59.233 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:46:59.233 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:46:59.233 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:46:59.233 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:46:59.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:46:59.507 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:46:59.499+0000 7fb9dfeeea40 -1 Falling back to public interface 2026-02-13T02:47:00.180 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:00.171+0000 7fb9dfeeea40 -1 osd.2 17 log_to_monitors true 2026-02-13T02:47:00.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:47:00.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-13T02:47:00.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:47:00.392 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:47:00.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:47:00.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:47:00.557 INFO:tasks.workunit.client.0.trial112.stderr:osd.2 up in weight 1 up_from 21 up_thru 0 down_at 18 last_clean_interval [14,17) [v2:127.0.0.1:6818/2189969289,v1:127.0.0.1:6819/2189969289] [v2:127.0.0.1:6820/2189969289,v1:127.0.0.1:6821/2189969289] exists,up 9e0a9018-0d7e-46cb-9a96-1a17a91de220 2026-02-13T02:47:00.557 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:47:00.557 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:47:00.557 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:47:00.557 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:47:00.557 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:47:00.557 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:47:00.557 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:47:00.558 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:47:00.558 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:47:00.559 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:47:00.559 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:47:00.559 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:47:00.559 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:47:00.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:47:00.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:47:00.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:47:00.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:47:00.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:47:00.664 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:47:00.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:47:00.829 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:47:00.829 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:47:00.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:47:00.829 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:47:00.830 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:47:00.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836498 2026-02-13T02:47:00.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836498 2026-02-13T02:47:00.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836498' 2026-02-13T02:47:00.900 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:47:00.901 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:47:00.971 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672975 2026-02-13T02:47:00.971 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672975 2026-02-13T02:47:00.971 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836498 1-42949672975' 2026-02-13T02:47:00.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:47:00.972 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:47:01.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=90194313218 2026-02-13T02:47:01.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 90194313218 2026-02-13T02:47:01.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836498 1-42949672975 2-90194313218' 2026-02-13T02:47:01.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:47:01.045 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836498 2026-02-13T02:47:01.045 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:47:01.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:47:01.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836498 2026-02-13T02:47:01.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:47:01.048 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836498 2026-02-13T02:47:01.049 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2026-02-13T02:47:01.049 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 21474836498 2026-02-13T02:47:01.049 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:47:01.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836496 -lt 21474836498 2026-02-13T02:47:01.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:47:02.214 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:47:02.215 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:47:02.376 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836499 -lt 21474836498 2026-02-13T02:47:02.376 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:47:02.377 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672975 2026-02-13T02:47:02.377 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:47:02.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:47:02.379 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672975 2026-02-13T02:47:02.379 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:47:02.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672975 2026-02-13T02:47:02.380 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2026-02-13T02:47:02.380 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 42949672975 2026-02-13T02:47:02.381 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:47:02.551 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672976 -lt 42949672975 2026-02-13T02:47:02.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:47:02.552 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-90194313218 2026-02-13T02:47:02.552 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:47:02.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:47:02.554 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-90194313218 2026-02-13T02:47:02.554 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:47:02.556 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=90194313218 2026-02-13T02:47:02.556 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 90194313218' 2026-02-13T02:47:02.556 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 90194313218 2026-02-13T02:47:02.556 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:47:02.721 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 90194313219 -lt 90194313218 2026-02-13T02:47:02.721 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:47:02.722 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:47:02.722 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:47:02.913 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:47:02.913 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:47:02.914 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:47:02.914 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:47:02.914 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:47:02.914 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:47:02.914 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:47:02.914 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:47:03.068 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:47:03.069 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:47:03.069 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:47:03.070 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:47:03.256 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:47:03.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:47:03.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:47:03.257 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:90: TEST_scrub_test: rm -f /tmp/tmp.6GDRsWdl1y 2026-02-13T02:47:03.258 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:92: TEST_scrub_test: local pgid=1.0 2026-02-13T02:47:03.258 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:93: TEST_scrub_test: pg_deep_scrub 1.0 2026-02-13T02:47:03.258 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1932: pg_deep_scrub: local pgid=1.0 2026-02-13T02:47:03.258 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: pg_deep_scrub: wait_for_pg_clean 1.0 2026-02-13T02:47:03.258 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=1.0 2026-02-13T02:47:03.258 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-13T02:47:03.259 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:47:03.259 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:47:03.259 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:47:03.259 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:47:03.259 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:47:03.512 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-13T02:47:03.513 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-13T02:47:03.513 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-13T02:47:03.513 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-13T02:47:03.513 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:47:03.513 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:47:03.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:47:03.673 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:47:03.673 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:47:03.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:47:03.673 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:47:03.673 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:47:03.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836502 2026-02-13T02:47:03.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836502 2026-02-13T02:47:03.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502' 2026-02-13T02:47:03.746 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:47:03.746 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:47:03.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672979 2026-02-13T02:47:03.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672979 2026-02-13T02:47:03.817 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-42949672979' 2026-02-13T02:47:03.818 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:47:03.818 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:47:03.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=90194313222 2026-02-13T02:47:03.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 90194313222 2026-02-13T02:47:03.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-42949672979 2-90194313222' 2026-02-13T02:47:03.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:47:03.890 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836502 2026-02-13T02:47:03.890 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:47:03.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:47:03.892 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836502 2026-02-13T02:47:03.892 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:47:03.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836502 2026-02-13T02:47:03.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2026-02-13T02:47:03.894 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836502 2026-02-13T02:47:03.895 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:47:04.049 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836502 -lt 21474836502 2026-02-13T02:47:04.049 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:47:04.050 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672979 2026-02-13T02:47:04.050 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:47:04.051 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:47:04.052 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672979 2026-02-13T02:47:04.052 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:47:04.054 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672979 2026-02-13T02:47:04.054 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 42949672979 2026-02-13T02:47:04.054 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2026-02-13T02:47:04.055 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:47:04.218 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672978 -lt 42949672979 2026-02-13T02:47:04.218 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:47:05.219 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:47:05.220 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:47:05.377 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672978 -lt 42949672979 2026-02-13T02:47:05.378 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:47:06.379 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 499 -eq 0 ']' 2026-02-13T02:47:06.379 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:47:06.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672981 -lt 42949672979 2026-02-13T02:47:06.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:47:06.543 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-90194313222 2026-02-13T02:47:06.543 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:47:06.544 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:47:06.545 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-90194313222 2026-02-13T02:47:06.546 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:47:06.547 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=90194313222 2026-02-13T02:47:06.547 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 90194313222' 2026-02-13T02:47:06.547 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 90194313222 2026-02-13T02:47:06.548 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:47:06.715 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 90194313224 -lt 90194313222 2026-02-13T02:47:06.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-13T02:47:06.716 INFO:tasks.workunit.client.0.trial112.stdout:#---------- 1.0 loop 0 2026-02-13T02:47:06.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 0' 2026-02-13T02:47:06.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-13T02:47:06.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-13T02:47:06.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-13T02:47:06.717 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-13T02:47:06.717 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-13T02:47:06.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-13T02:47:06.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-13T02:47:06.787 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-13T02:47:06.788 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-13T02:47:06.788 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2026-02-13T02:47:06.788 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:47:06.788 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:47:06.789 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:47:06.789 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2026-02-13T02:47:06.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: local last_scrub=2026-02-13T02:46:52.578146+0000 2026-02-13T02:47:06.942 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: pg_deep_scrub: ceph pg deep-scrub 1.0 2026-02-13T02:47:07.095 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 1.0 on osd.1 to deep-scrub 2026-02-13T02:47:07.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: pg_deep_scrub: wait_for_scrub 1.0 2026-02-13T02:46:52.578146+0000 last_deep_scrub_stamp 2026-02-13T02:47:07.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-13T02:47:07.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:46:52.578146+0000 2026-02-13T02:47:07.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_deep_scrub_stamp 2026-02-13T02:47:07.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:47:07.106 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:47:07.106 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2026-02-13T02:47:07.106 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:47:07.106 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:47:07.107 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:47:07.107 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2026-02-13T02:47:07.269 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:46:52.578146+0000 '>' 2026-02-13T02:46:52.578146+0000 2026-02-13T02:47:07.269 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:47:08.270 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:47:08.270 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:47:08.271 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2026-02-13T02:47:08.271 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:47:08.271 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:47:08.272 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:47:08.272 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2026-02-13T02:47:08.430 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:46:52.578146+0000 '>' 2026-02-13T02:46:52.578146+0000 2026-02-13T02:47:08.431 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:47:09.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:47:09.432 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:47:09.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2026-02-13T02:47:09.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:47:09.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:47:09.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2026-02-13T02:47:09.433 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:47:09.596 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:46:52.578146+0000 '>' 2026-02-13T02:46:52.578146+0000 2026-02-13T02:47:09.596 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:47:10.597 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:47:10.597 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:47:10.598 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2026-02-13T02:47:10.598 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:47:10.598 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:47:10.598 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:47:10.598 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2026-02-13T02:47:10.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:47:07.946174+0000 '>' 2026-02-13T02:46:52.578146+0000 2026-02-13T02:47:10.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:47:10.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:95: TEST_scrub_test: ceph pg dump pgs 2026-02-13T02:47:10.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:95: TEST_scrub_test: grep '^1.0' 2026-02-13T02:47:10.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:95: TEST_scrub_test: grep -q -- +inconsistent 2026-02-13T02:47:10.905 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:47:10.917 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:96: TEST_scrub_test: ceph pg 1.0 query 2026-02-13T02:47:10.917 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:96: TEST_scrub_test: jq .info.stats.stat_sum.num_scrub_errors 2026-02-13T02:47:10.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:96: TEST_scrub_test: test 2 = 2 2026-02-13T02:47:10.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:98: TEST_scrub_test: ceph osd out 1 2026-02-13T02:47:11.193 INFO:tasks.workunit.client.0.trial112.stderr:marked out osd.1. 2026-02-13T02:47:11.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:99: TEST_scrub_test: wait_for_clean 2026-02-13T02:47:11.208 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:47:11.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:47:11.209 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:47:11.210 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:47:11.210 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:47:11.210 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:47:11.210 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:47:11.210 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:47:11.210 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:47:11.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:47:11.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:47:11.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:47:11.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:47:11.320 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:47:11.320 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:47:11.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:47:11.484 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:47:11.484 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:47:11.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:47:11.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:47:11.484 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:47:11.557 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836511 2026-02-13T02:47:11.558 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836511 2026-02-13T02:47:11.558 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511' 2026-02-13T02:47:11.558 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:47:11.558 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:47:11.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672988 2026-02-13T02:47:11.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672988 2026-02-13T02:47:11.628 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511 1-42949672988' 2026-02-13T02:47:11.629 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:47:11.629 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:47:11.704 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=90194313231 2026-02-13T02:47:11.704 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 90194313231 2026-02-13T02:47:11.705 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511 1-42949672988 2-90194313231' 2026-02-13T02:47:11.705 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:47:11.705 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836511 2026-02-13T02:47:11.705 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:47:11.706 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:47:11.707 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836511 2026-02-13T02:47:11.707 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:47:11.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836511 2026-02-13T02:47:11.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2026-02-13T02:47:11.709 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836511 2026-02-13T02:47:11.709 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:47:11.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836511 -lt 21474836511 2026-02-13T02:47:11.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:47:11.882 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672988 2026-02-13T02:47:11.882 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:47:11.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:47:11.884 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672988 2026-02-13T02:47:11.885 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:47:11.886 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672988 2026-02-13T02:47:11.886 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672988' 2026-02-13T02:47:11.887 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 42949672988 2026-02-13T02:47:11.887 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:47:12.042 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672988 -lt 42949672988 2026-02-13T02:47:12.042 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:47:12.043 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-90194313231 2026-02-13T02:47:12.043 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:47:12.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:47:12.045 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-90194313231 2026-02-13T02:47:12.045 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:47:12.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=90194313231 2026-02-13T02:47:12.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 90194313231' 2026-02-13T02:47:12.047 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 90194313231 2026-02-13T02:47:12.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:47:12.212 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 90194313231 -lt 90194313231 2026-02-13T02:47:12.213 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:47:12.213 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:47:12.213 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:47:12.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:47:12.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:47:12.406 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:47:12.406 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:47:12.406 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:47:12.406 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:47:12.406 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:47:12.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:47:12.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:47:12.567 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:47:12.568 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:47:12.568 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:47:12.766 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:47:12.767 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:47:12.767 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:47:12.767 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:101: TEST_scrub_test: pg_deep_scrub 1.0 2026-02-13T02:47:12.767 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1932: pg_deep_scrub: local pgid=1.0 2026-02-13T02:47:12.767 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: pg_deep_scrub: wait_for_pg_clean 1.0 2026-02-13T02:47:12.767 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=1.0 2026-02-13T02:47:12.767 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-13T02:47:12.768 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:47:12.768 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:47:12.768 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:47:12.768 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:47:12.769 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:47:13.049 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-13T02:47:13.049 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-13T02:47:13.049 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-13T02:47:13.049 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-13T02:47:13.049 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:47:13.049 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:47:13.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:47:13.204 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:47:13.204 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:47:13.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:47:13.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:47:13.205 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:47:13.277 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836514 2026-02-13T02:47:13.277 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836514 2026-02-13T02:47:13.277 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514' 2026-02-13T02:47:13.277 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:47:13.277 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:47:13.347 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672991 2026-02-13T02:47:13.347 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672991 2026-02-13T02:47:13.347 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514 1-42949672991' 2026-02-13T02:47:13.348 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:47:13.348 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:47:13.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=90194313234 2026-02-13T02:47:13.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 90194313234 2026-02-13T02:47:13.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514 1-42949672991 2-90194313234' 2026-02-13T02:47:13.420 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:47:13.421 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836514 2026-02-13T02:47:13.421 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:47:13.422 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:47:13.423 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836514 2026-02-13T02:47:13.423 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:47:13.424 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836514 2026-02-13T02:47:13.424 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836514 2026-02-13T02:47:13.425 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2026-02-13T02:47:13.425 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:47:13.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836511 -lt 21474836514 2026-02-13T02:47:13.588 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:47:14.590 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:47:14.590 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:47:14.749 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836514 -lt 21474836514 2026-02-13T02:47:14.750 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:47:14.750 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672991 2026-02-13T02:47:14.750 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:47:14.752 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:47:14.752 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672991 2026-02-13T02:47:14.753 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:47:14.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672991 2026-02-13T02:47:14.754 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 42949672991 2026-02-13T02:47:14.755 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672991' 2026-02-13T02:47:14.755 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:47:14.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672991 -lt 42949672991 2026-02-13T02:47:14.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:47:14.918 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-90194313234 2026-02-13T02:47:14.918 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:47:14.919 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:47:14.921 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-90194313234 2026-02-13T02:47:14.921 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:47:14.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=90194313234 2026-02-13T02:47:14.922 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 90194313234 2026-02-13T02:47:14.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 90194313234' 2026-02-13T02:47:14.923 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:47:15.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 90194313234 -lt 90194313234 2026-02-13T02:47:15.087 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-13T02:47:15.087 INFO:tasks.workunit.client.0.trial112.stdout:#---------- 1.0 loop 0 2026-02-13T02:47:15.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 0' 2026-02-13T02:47:15.088 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-13T02:47:15.089 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-13T02:47:15.089 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-13T02:47:15.089 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-13T02:47:15.089 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-13T02:47:15.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean+remapped 2026-02-13T02:47:15.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean+remapped == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-13T02:47:15.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-13T02:47:15.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-13T02:47:15.165 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2026-02-13T02:47:15.165 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:47:15.165 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:47:15.165 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:47:15.165 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2026-02-13T02:47:15.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: local last_scrub=2026-02-13T02:47:07.946174+0000 2026-02-13T02:47:15.318 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: pg_deep_scrub: ceph pg deep-scrub 1.0 2026-02-13T02:47:15.470 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 1.0 on osd.0 to deep-scrub 2026-02-13T02:47:15.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: pg_deep_scrub: wait_for_scrub 1.0 2026-02-13T02:47:07.946174+0000 last_deep_scrub_stamp 2026-02-13T02:47:15.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-13T02:47:15.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:47:07.946174+0000 2026-02-13T02:47:15.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_deep_scrub_stamp 2026-02-13T02:47:15.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:47:15.481 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:47:15.481 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2026-02-13T02:47:15.481 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:47:15.481 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:47:15.481 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:47:15.482 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2026-02-13T02:47:15.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:47:07.946174+0000 '>' 2026-02-13T02:47:07.946174+0000 2026-02-13T02:47:15.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:47:16.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:47:16.642 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:47:16.643 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2026-02-13T02:47:16.643 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:47:16.643 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:47:16.643 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:47:16.643 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2026-02-13T02:47:16.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:47:07.946174+0000 '>' 2026-02-13T02:47:07.946174+0000 2026-02-13T02:47:16.808 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:47:17.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:47:17.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:47:17.810 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2026-02-13T02:47:17.810 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:47:17.810 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-13T02:47:17.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2026-02-13T02:47:17.811 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:47:17.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:47:15.702623+0000 '>' 2026-02-13T02:47:07.946174+0000 2026-02-13T02:47:17.972 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:47:17.973 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:103: TEST_scrub_test: ceph pg 1.0 query 2026-02-13T02:47:17.973 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:103: TEST_scrub_test: jq .info.stats.stat_sum.num_scrub_errors 2026-02-13T02:47:18.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:103: TEST_scrub_test: test 2 = 2 2026-02-13T02:47:18.046 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:104: TEST_scrub_test: ceph pg 1.0 query 2026-02-13T02:47:18.046 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:104: TEST_scrub_test: jq '.peer_info[0].stats.stat_sum.num_scrub_errors' 2026-02-13T02:47:18.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:104: TEST_scrub_test: test 2 = 2 2026-02-13T02:47:18.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:105: TEST_scrub_test: ceph pg dump pgs 2026-02-13T02:47:18.118 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:105: TEST_scrub_test: grep '^1.0' 2026-02-13T02:47:18.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:105: TEST_scrub_test: grep -q -- +inconsistent 2026-02-13T02:47:18.266 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:47:18.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:107: TEST_scrub_test: ceph osd in 1 2026-02-13T02:47:18.475 INFO:tasks.workunit.client.0.trial112.stderr:marked in osd.1. 2026-02-13T02:47:18.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:108: TEST_scrub_test: wait_for_clean 2026-02-13T02:47:18.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:47:18.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:47:18.489 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:47:18.490 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:47:18.490 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:47:18.490 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:47:18.490 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:47:18.490 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:47:18.490 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:47:18.611 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:47:18.611 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:47:18.611 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:47:18.611 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:47:18.611 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:47:18.611 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:47:18.769 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:47:18.769 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:47:18.769 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:47:18.769 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:47:18.769 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:47:18.770 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:47:18.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836520 2026-02-13T02:47:18.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836520 2026-02-13T02:47:18.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836520' 2026-02-13T02:47:18.841 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:47:18.841 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:47:18.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672997 2026-02-13T02:47:18.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672997 2026-02-13T02:47:18.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836520 1-42949672997' 2026-02-13T02:47:18.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:47:18.916 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:47:18.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=90194313240 2026-02-13T02:47:18.987 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 90194313240 2026-02-13T02:47:18.988 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836520 1-42949672997 2-90194313240' 2026-02-13T02:47:18.988 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:47:18.988 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836520 2026-02-13T02:47:18.988 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:47:18.989 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:47:18.990 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836520 2026-02-13T02:47:18.990 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:47:18.992 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836520 2026-02-13T02:47:18.992 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836520 2026-02-13T02:47:18.993 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836520' 2026-02-13T02:47:18.993 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:47:19.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836518 -lt 21474836520 2026-02-13T02:47:19.164 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:47:20.165 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:47:20.166 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:47:20.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836521 -lt 21474836520 2026-02-13T02:47:20.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:47:20.329 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672997 2026-02-13T02:47:20.329 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:47:20.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:47:20.331 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672997 2026-02-13T02:47:20.331 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:47:20.332 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672997 2026-02-13T02:47:20.332 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672997' 2026-02-13T02:47:20.333 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 42949672997 2026-02-13T02:47:20.333 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:47:20.508 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672998 -lt 42949672997 2026-02-13T02:47:20.508 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:47:20.509 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-90194313240 2026-02-13T02:47:20.509 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:47:20.510 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:47:20.511 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-90194313240 2026-02-13T02:47:20.511 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:47:20.513 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=90194313240 2026-02-13T02:47:20.513 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 90194313240' 2026-02-13T02:47:20.513 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 90194313240 2026-02-13T02:47:20.513 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:47:20.682 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 90194313241 -lt 90194313240 2026-02-13T02:47:20.683 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:47:20.683 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:47:20.683 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:47:20.876 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:47:20.876 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:47:20.877 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:47:20.877 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:47:20.877 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:47:20.877 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:47:20.877 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:47:20.877 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:47:21.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:47:21.033 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:47:21.034 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:47:21.034 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:47:21.230 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:47:21.230 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:47:21.230 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:47:21.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:110: TEST_scrub_test: repair 1.0 2026-02-13T02:47:21.231 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=1.0 2026-02-13T02:47:21.231 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 1.0 2026-02-13T02:47:21.231 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:47:21.231 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:47:21.231 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:47:21.232 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:47:21.393 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-13T02:47:15.702623+0000 2026-02-13T02:47:21.393 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 1.0 2026-02-13T02:47:21.541 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 1.0 on osd.1 to repair 2026-02-13T02:47:21.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 1.0 2026-02-13T02:47:15.702623+0000 2026-02-13T02:47:21.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-13T02:47:21.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-13T02:47:15.702623+0000 2026-02-13T02:47:21.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-13T02:47:21.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-13T02:47:21.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:47:21.554 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:47:21.554 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:47:21.555 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:47:21.555 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:47:21.555 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:47:21.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:47:15.702623+0000 '>' 2026-02-13T02:47:15.702623+0000 2026-02-13T02:47:21.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:47:22.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:47:22.717 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:47:22.718 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:47:22.718 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:47:22.718 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:47:22.718 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:47:22.719 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:47:22.879 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:47:15.702623+0000 '>' 2026-02-13T02:47:15.702623+0000 2026-02-13T02:47:22.880 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-13T02:47:23.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-13T02:47:23.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-13T02:47:23.881 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-13T02:47:23.881 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-13T02:47:23.881 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-13T02:47:23.882 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-13T02:47:23.882 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-13T02:47:24.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-13T02:47:22.953343+0000 '>' 2026-02-13T02:47:15.702623+0000 2026-02-13T02:47:24.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-13T02:47:24.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:111: TEST_scrub_test: wait_for_clean 2026-02-13T02:47:24.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:47:24.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:47:24.038 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:47:24.039 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:47:24.039 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:47:24.039 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:47:24.040 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:47:24.040 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:47:24.040 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:47:24.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:47:24.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:47:24.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:47:24.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:47:24.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:47:24.151 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:47:24.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:47:24.317 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:47:24.317 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:47:24.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:47:24.317 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:47:24.318 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:47:24.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836527 2026-02-13T02:47:24.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836527 2026-02-13T02:47:24.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836527' 2026-02-13T02:47:24.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:47:24.390 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:47:24.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673004 2026-02-13T02:47:24.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673004 2026-02-13T02:47:24.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836527 1-42949673004' 2026-02-13T02:47:24.466 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:47:24.466 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:47:24.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=90194313247 2026-02-13T02:47:24.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 90194313247 2026-02-13T02:47:24.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836527 1-42949673004 2-90194313247' 2026-02-13T02:47:24.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:47:24.542 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836527 2026-02-13T02:47:24.542 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:47:24.543 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:47:24.544 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836527 2026-02-13T02:47:24.544 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:47:24.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836527 2026-02-13T02:47:24.546 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836527' 2026-02-13T02:47:24.546 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836527 2026-02-13T02:47:24.547 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:47:24.705 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836525 -lt 21474836527 2026-02-13T02:47:24.705 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:47:25.707 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-13T02:47:25.707 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:47:25.872 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836528 -lt 21474836527 2026-02-13T02:47:25.873 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:47:25.873 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673004 2026-02-13T02:47:25.873 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:47:25.874 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:47:25.875 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673004 2026-02-13T02:47:25.875 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:47:25.877 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673004 2026-02-13T02:47:25.877 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673004' 2026-02-13T02:47:25.877 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 42949673004 2026-02-13T02:47:25.877 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:47:26.047 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673005 -lt 42949673004 2026-02-13T02:47:26.047 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:47:26.048 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-90194313247 2026-02-13T02:47:26.048 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:47:26.049 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:47:26.050 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-90194313247 2026-02-13T02:47:26.050 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:47:26.052 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=90194313247 2026-02-13T02:47:26.052 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 90194313247' 2026-02-13T02:47:26.052 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 90194313247 2026-02-13T02:47:26.052 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:47:26.215 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 90194313248 -lt 90194313247 2026-02-13T02:47:26.215 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:47:26.216 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:47:26.216 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:47:26.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:47:26.412 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:47:26.413 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:47:26.413 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:47:26.413 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:47:26.413 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:47:26.413 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:47:26.414 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:47:26.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:47:26.570 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:47:26.570 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:47:26.570 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:47:26.768 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:47:26.768 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:47:26.768 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:47:26.769 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:114: TEST_scrub_test: ceph pg 1.0 query 2026-02-13T02:47:26.769 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:114: TEST_scrub_test: jq '.peer_info[0].stats.stat_sum.num_scrub_errors' 2026-02-13T02:47:26.842 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:114: TEST_scrub_test: test 2 = 2 2026-02-13T02:47:26.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:115: TEST_scrub_test: ceph pg dump pgs 2026-02-13T02:47:26.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:115: TEST_scrub_test: grep '^1.0' 2026-02-13T02:47:26.843 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:115: TEST_scrub_test: grep -vq -- +inconsistent 2026-02-13T02:47:26.990 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:47:26.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:117: TEST_scrub_test: ceph osd out 1 2026-02-13T02:47:27.207 INFO:tasks.workunit.client.0.trial112.stderr:marked out osd.1. 2026-02-13T02:47:27.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:118: TEST_scrub_test: wait_for_clean 2026-02-13T02:47:27.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:47:27.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:47:27.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:47:27.222 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:47:27.222 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:47:27.222 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:47:27.222 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:47:27.222 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:47:27.222 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:47:27.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:47:27.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:47:27.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:47:27.329 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:47:27.330 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-13T02:47:27.330 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:47:27.491 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:47:27.491 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:47:27.491 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:47:27.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:47:27.492 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:47:27.492 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:47:27.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836531 2026-02-13T02:47:27.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836531 2026-02-13T02:47:27.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836531' 2026-02-13T02:47:27.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:47:27.567 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:47:27.640 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673008 2026-02-13T02:47:27.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673008 2026-02-13T02:47:27.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836531 1-42949673008' 2026-02-13T02:47:27.641 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:47:27.641 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:47:27.711 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=90194313251 2026-02-13T02:47:27.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 90194313251 2026-02-13T02:47:27.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836531 1-42949673008 2-90194313251' 2026-02-13T02:47:27.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:47:27.712 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836531 2026-02-13T02:47:27.713 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:47:27.714 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:47:27.714 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836531 2026-02-13T02:47:27.715 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:47:27.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836531 2026-02-13T02:47:27.716 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836531' 2026-02-13T02:47:27.716 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836531 2026-02-13T02:47:27.717 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:47:27.882 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836531 -lt 21474836531 2026-02-13T02:47:27.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:47:27.883 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673008 2026-02-13T02:47:27.883 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:47:27.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:47:27.885 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673008 2026-02-13T02:47:27.885 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:47:27.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673008 2026-02-13T02:47:27.887 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 42949673008 2026-02-13T02:47:27.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673008' 2026-02-13T02:47:27.887 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:47:28.057 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673008 -lt 42949673008 2026-02-13T02:47:28.057 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:47:28.058 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-90194313251 2026-02-13T02:47:28.058 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:47:28.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:47:28.060 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-90194313251 2026-02-13T02:47:28.060 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:47:28.061 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=90194313251 2026-02-13T02:47:28.062 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 90194313251 2026-02-13T02:47:28.062 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 90194313251' 2026-02-13T02:47:28.062 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:47:28.220 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 90194313251 -lt 90194313251 2026-02-13T02:47:28.221 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:47:28.222 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:47:28.222 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:47:28.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:47:28.417 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:47:28.417 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:47:28.417 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:47:28.417 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:47:28.417 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:47:28.418 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:47:28.418 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:47:28.580 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:47:28.580 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:47:28.581 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:47:28.581 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:47:28.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:47:28.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:47:28.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:47:28.776 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:120: TEST_scrub_test: ceph pg 1.0 query 2026-02-13T02:47:28.777 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:120: TEST_scrub_test: jq .info.stats.stat_sum.num_scrub_errors 2026-02-13T02:47:28.846 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:120: TEST_scrub_test: test 0 = 0 2026-02-13T02:47:28.847 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:121: TEST_scrub_test: ceph pg 1.0 query 2026-02-13T02:47:28.847 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:121: TEST_scrub_test: jq '.peer_info[0].stats.stat_sum.num_scrub_errors' 2026-02-13T02:47:28.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:121: TEST_scrub_test: test 0 = 0 2026-02-13T02:47:28.922 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:122: TEST_scrub_test: ceph pg 1.0 query 2026-02-13T02:47:28.922 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:122: TEST_scrub_test: jq '.peer_info[1].stats.stat_sum.num_scrub_errors' 2026-02-13T02:47:28.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:122: TEST_scrub_test: test 0 = 0 2026-02-13T02:47:28.994 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:123: TEST_scrub_test: ceph pg dump pgs 2026-02-13T02:47:28.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:123: TEST_scrub_test: grep '^1.0' 2026-02-13T02:47:28.995 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:123: TEST_scrub_test: grep -vq -- +inconsistent 2026-02-13T02:47:29.144 INFO:tasks.workunit.client.0.trial112.stderr:dumped pgs 2026-02-13T02:47:29.154 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:124: TEST_scrub_test: perf_counters td/osd-scrub-test 3 2026-02-13T02:47:29.154 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2026-02-13T02:47:29.154 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=3 2026-02-13T02:47:29.154 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 3 - 1 2026-02-13T02:47:29.156 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 2 2026-02-13T02:47:29.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:47:29.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2026-02-13T02:47:29.157 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-13T02:47:29.238 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:47:29.238 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_ec": [ 2026-02-13T02:47:29.238 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:47:29.238 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:47:29.239 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:47:29.239 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:47:29.239 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:47:29.239 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:47:29.239 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:47:29.239 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:47:29.239 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:47:29.239 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:47:29.239 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:47:29.239 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:47:29.239 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:47:29.239 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:47:29.240 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_repl": [ 2026-02-13T02:47:29.240 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:47:29.240 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:47:29.240 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:47:29.240 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:47:29.240 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:47:29.240 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:47:29.240 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:47:29.240 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 2, 2026-02-13T02:47:29.240 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:47:29.240 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:47:29.240 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:47:29.240 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:47:29.241 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:47:29.241 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:47:29.241 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_ec": [ 2026-02-13T02:47:29.241 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:47:29.241 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:47:29.241 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:47:29.241 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:47:29.241 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:47:29.241 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:47:29.241 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:47:29.241 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:47:29.241 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:47:29.241 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:47:29.242 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:47:29.242 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:47:29.242 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:47:29.242 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:47:29.242 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_repl": [ 2026-02-13T02:47:29.242 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:47:29.242 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:47:29.242 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:47:29.242 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:47:29.242 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:47:29.242 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:47:29.242 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:47:29.242 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:47:29.243 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:47:29.243 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:47:29.243 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:47:29.243 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:47:29.243 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:47:29.243 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:47:29.243 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:47:29.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:47:29.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2026-02-13T02:47:29.244 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-13T02:47:29.309 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:47:29.309 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_ec": [ 2026-02-13T02:47:29.309 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:47:29.309 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:47:29.309 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:47:29.309 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:47:29.309 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:47:29.309 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:47:29.309 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:47:29.310 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:47:29.310 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:47:29.310 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:47:29.310 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:47:29.310 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:47:29.310 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:47:29.310 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:47:29.310 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_repl": [ 2026-02-13T02:47:29.310 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:47:29.310 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:47:29.310 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:47:29.310 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:47:29.311 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:47:29.311 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:47:29.311 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:47:29.311 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 6, 2026-02-13T02:47:29.311 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:47:29.311 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:47:29.311 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:47:29.311 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:47:29.311 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:47:29.311 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:47:29.311 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_ec": [ 2026-02-13T02:47:29.311 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:47:29.311 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:47:29.312 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:47:29.312 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:47:29.312 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:47:29.312 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:47:29.312 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:47:29.312 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:47:29.312 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:47:29.312 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:47:29.312 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:47:29.312 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:47:29.312 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:47:29.312 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:47:29.312 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_repl": [ 2026-02-13T02:47:29.312 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:47:29.313 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:47:29.313 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:47:29.313 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:47:29.313 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:47:29.313 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:47:29.313 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:47:29.313 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:47:29.313 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:47:29.313 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:47:29.313 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:47:29.313 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:47:29.313 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:47:29.313 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:47:29.314 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:47:29.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-13T02:47:29.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.2 counter dump 2026-02-13T02:47:29.314 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-13T02:47:29.384 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:47:29.384 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_ec": [ 2026-02-13T02:47:29.384 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:47:29.384 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:47:29.384 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:47:29.384 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:47:29.384 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:47:29.385 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:47:29.385 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:47:29.385 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:47:29.385 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:47:29.385 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:47:29.385 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:47:29.385 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:47:29.385 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:47:29.385 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:47:29.385 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_dp_repl": [ 2026-02-13T02:47:29.385 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:47:29.385 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:47:29.386 INFO:tasks.workunit.client.0.trial112.stdout: "level": "deep", 2026-02-13T02:47:29.386 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:47:29.386 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:47:29.386 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:47:29.386 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:47:29.386 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:47:29.386 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:47:29.386 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:47:29.386 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:47:29.386 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:47:29.386 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:47:29.386 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:47:29.386 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_ec": [ 2026-02-13T02:47:29.387 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:47:29.387 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:47:29.387 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:47:29.387 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "ec" 2026-02-13T02:47:29.387 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:47:29.387 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:47:29.387 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:47:29.387 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:47:29.387 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:47:29.387 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:47:29.387 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:47:29.387 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:47:29.387 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:47:29.388 INFO:tasks.workunit.client.0.trial112.stdout: ], 2026-02-13T02:47:29.388 INFO:tasks.workunit.client.0.trial112.stdout: "osd_scrub_sh_repl": [ 2026-02-13T02:47:29.388 INFO:tasks.workunit.client.0.trial112.stdout: { 2026-02-13T02:47:29.388 INFO:tasks.workunit.client.0.trial112.stdout: "labels": { 2026-02-13T02:47:29.388 INFO:tasks.workunit.client.0.trial112.stdout: "level": "shallow", 2026-02-13T02:47:29.388 INFO:tasks.workunit.client.0.trial112.stdout: "pooltype": "replicated" 2026-02-13T02:47:29.388 INFO:tasks.workunit.client.0.trial112.stdout: }, 2026-02-13T02:47:29.388 INFO:tasks.workunit.client.0.trial112.stdout: "counters": { 2026-02-13T02:47:29.388 INFO:tasks.workunit.client.0.trial112.stdout: "preemptions": 0, 2026-02-13T02:47:29.388 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_selected": 0, 2026-02-13T02:47:29.388 INFO:tasks.workunit.client.0.trial112.stdout: "chunk_busy": 0, 2026-02-13T02:47:29.388 INFO:tasks.workunit.client.0.trial112.stdout: "locked_object": 0, 2026-02-13T02:47:29.389 INFO:tasks.workunit.client.0.trial112.stdout: "write_blocked_by_scrub": 0 2026-02-13T02:47:29.389 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:47:29.389 INFO:tasks.workunit.client.0.trial112.stdout: } 2026-02-13T02:47:29.389 INFO:tasks.workunit.client.0.trial112.stdout: ] 2026-02-13T02:47:29.389 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:47:29.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_scrub_test ------------------' 2026-02-13T02:47:29.390 INFO:tasks.workunit.client.0.trial112.stdout:-------------- Teardown Test TEST_scrub_test ------------------ 2026-02-13T02:47:29.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2026-02-13T02:47:29.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-13T02:47:29.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:47:29.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-13T02:47:29.391 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:47:29.391 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:47:29.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:47:29.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:47:29.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:47:29.496 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:47:29.496 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:47:29.497 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:47:29.498 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:47:29.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:47:29.499 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:47:29.500 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:47:29.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:47:29.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:47:29.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:47:29.502 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:47:29.502 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:47:29.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:47:29.503 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-13T02:47:29.518 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:47:29.518 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:47:29.518 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:47:29.518 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.508946 2026-02-13T02:47:29.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:47:29.519 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:47:29.520 INFO:tasks.workunit.client.0.trial112.stdout:-------------- Complete Test TEST_scrub_test ------------------ 2026-02-13T02:47:29.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_scrub_test ------------------' 2026-02-13T02:47:29.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: main: code=0 2026-02-13T02:47:29.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: teardown td/osd-scrub-test 0 2026-02-13T02:47:29.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-13T02:47:29.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-13T02:47:29.521 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-13T02:47:29.521 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:47:29.522 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:47:29.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:47:29.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:47:29.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:47:29.522 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:47:29.523 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:47:29.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:47:29.524 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:47:29.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:47:29.526 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:47:29.526 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:47:29.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:47:29.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:47:29.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:47:29.528 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:47:29.530 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:47:29.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-13T02:47:29.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-13T02:47:29.532 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:47:29.533 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:47:29.533 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.508946 2026-02-13T02:47:29.533 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.508946 2026-02-13T02:47:29.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:47:29.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:47:29.534 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: main: return 0 2026-02-13T02:47:29.535 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-13T02:47:29.536 DEBUG:teuthology.orchestra.run.trial112:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-13T02:47:29.589 INFO:tasks.workunit:Running workunit scrub/osd-unexpected-clone.sh... 2026-02-13T02:47:29.590 DEBUG:teuthology.orchestra.run.trial112:workunit test scrub/osd-unexpected-clone.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3b40640f18f2d34a761ee9b6dba2dfbb2a3118eb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh 2026-02-13T02:47:29.647 INFO:tasks.workunit.client.0.trial112.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-13T02:47:29.650 INFO:tasks.workunit.client.0.trial112.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-13T02:47:29.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-13T02:47:29.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-13T02:47:29.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export PYTHONWARNINGS=ignore 2026-02-13T02:47:29.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: PYTHONWARNINGS=ignore 2026-02-13T02:47:29.650 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: export CEPH_CONF=/dev/null 2026-02-13T02:47:29.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: CEPH_CONF=/dev/null 2026-02-13T02:47:29.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: main: unset CEPH_ARGS 2026-02-13T02:47:29.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: local code 2026-02-13T02:47:29.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: run td/osd-unexpected-clone 2026-02-13T02:47:29.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:22: run: local dir=td/osd-unexpected-clone 2026-02-13T02:47:29.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:23: run: shift 2026-02-13T02:47:29.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:25: run: export CEPH_MON=127.0.0.1:7144 2026-02-13T02:47:29.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:25: run: CEPH_MON=127.0.0.1:7144 2026-02-13T02:47:29.651 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:26: run: export CEPH_ARGS 2026-02-13T02:47:29.651 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:27: run: uuidgen 2026-02-13T02:47:29.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:27: run: CEPH_ARGS+='--fsid=61ed748e-0c83-461d-a19a-2a41771343bc --auth-supported=none ' 2026-02-13T02:47:29.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7144 ' 2026-02-13T02:47:29.652 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:30: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2026-02-13T02:47:29.652 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:31: run: set 2026-02-13T02:47:29.653 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-13T02:47:29.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:31: run: local funcs=TEST_recover_unexpected 2026-02-13T02:47:29.654 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:32: run: for func in $funcs 2026-02-13T02:47:29.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:33: run: setup td/osd-unexpected-clone 2026-02-13T02:47:29.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-unexpected-clone 2026-02-13T02:47:29.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-unexpected-clone 2026-02-13T02:47:29.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-unexpected-clone 2026-02-13T02:47:29.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:47:29.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-unexpected-clone KILL 2026-02-13T02:47:29.655 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:47:29.655 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:47:29.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:47:29.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:47:29.655 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:47:29.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:47:29.656 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:47:29.657 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:47:29.657 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:47:29.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:47:29.658 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:47:29.659 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:47:29.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:47:29.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:47:29.660 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:47:29.660 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:47:29.661 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:47:29.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:47:29.663 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-unexpected-clone 2026-02-13T02:47:29.664 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:47:29.664 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:47:29.664 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.653483 2026-02-13T02:47:29.664 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.653483 2026-02-13T02:47:29.665 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:47:29.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:47:29.666 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-unexpected-clone 2026-02-13T02:47:29.667 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-13T02:47:29.667 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:47:29.667 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.653483 2026-02-13T02:47:29.668 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.653483 2026-02-13T02:47:29.669 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-13T02:47:29.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-13T02:47:29.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-13T02:47:29.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-13T02:47:29.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-unexpected-clone 1' TERM HUP INT 2026-02-13T02:47:29.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:34: run: TEST_recover_unexpected td/osd-unexpected-clone 2026-02-13T02:47:29.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:40: TEST_recover_unexpected: local dir=td/osd-unexpected-clone 2026-02-13T02:47:29.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:42: TEST_recover_unexpected: run_mon td/osd-unexpected-clone a 2026-02-13T02:47:29.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-unexpected-clone 2026-02-13T02:47:29.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-13T02:47:29.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-13T02:47:29.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-13T02:47:29.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-unexpected-clone/a 2026-02-13T02:47:29.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-unexpected-clone/a --run-dir=td/osd-unexpected-clone 2026-02-13T02:47:29.691 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-13T02:47:29.691 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:47:29.692 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:47:29.692 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:47:29.692 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:47:29.692 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.653483 2026-02-13T02:47:29.692 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.653483/$cluster-$name.asok' 2026-02-13T02:47:29.692 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-unexpected-clone/a '--log-file=td/osd-unexpected-clone/$name.log' '--admin-socket=/tmp/ceph-asok.653483/$cluster-$name.asok' --mon-cluster-log-file=td/osd-unexpected-clone/log --run-dir=td/osd-unexpected-clone '--pid-file=td/osd-unexpected-clone/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-13T02:47:29.720 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-13T02:47:29.721 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-13T02:47:29.722 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:47:29.722 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:47:29.722 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-13T02:47:29.723 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-13T02:47:29.723 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:47:29.723 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:47:29.723 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:47:29.723 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:47:29.723 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:47:29.724 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.653483 2026-02-13T02:47:29.724 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.653483/ceph-mon.a.asok 2026-02-13T02:47:29.724 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:47:29.724 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.653483/ceph-mon.a.asok config get fsid 2026-02-13T02:47:29.779 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-13T02:47:29.779 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-13T02:47:29.779 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-13T02:47:29.779 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-13T02:47:29.779 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-13T02:47:29.780 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-13T02:47:29.780 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-13T02:47:29.780 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-13T02:47:29.781 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-13T02:47:29.781 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:47:29.781 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.653483 2026-02-13T02:47:29.781 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.653483/ceph-mon.a.asok 2026-02-13T02:47:29.781 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-13T02:47:29.781 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.653483/ceph-mon.a.asok config get mon_host 2026-02-13T02:47:29.835 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:43: TEST_recover_unexpected: run_mgr td/osd-unexpected-clone x 2026-02-13T02:47:29.835 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-unexpected-clone 2026-02-13T02:47:29.835 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-13T02:47:29.835 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-13T02:47:29.835 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-13T02:47:29.835 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-unexpected-clone/x 2026-02-13T02:47:29.835 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-13T02:47:29.929 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-13T02:47:29.929 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:47:29.929 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:47:29.929 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:47:29.929 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:47:29.929 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.653483 2026-02-13T02:47:29.930 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.653483/$cluster-$name.asok' 2026-02-13T02:47:29.930 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:47:29.931 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-unexpected-clone/x '--log-file=td/osd-unexpected-clone/$name.log' '--admin-socket=/tmp/ceph-asok.653483/$cluster-$name.asok' --run-dir=td/osd-unexpected-clone '--pid-file=td/osd-unexpected-clone/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-13T02:47:29.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:44: TEST_recover_unexpected: run_osd td/osd-unexpected-clone 0 2026-02-13T02:47:29.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-unexpected-clone 2026-02-13T02:47:29.952 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:47:29.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-13T02:47:29.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:47:29.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-unexpected-clone/0 2026-02-13T02:47:29.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=61ed748e-0c83-461d-a19a-2a41771343bc --auth-supported=none --mon-host=127.0.0.1:7144 ' 2026-02-13T02:47:29.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:47:29.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:47:29.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:47:29.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-unexpected-clone/0' 2026-02-13T02:47:29.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-unexpected-clone/0/journal' 2026-02-13T02:47:29.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:47:29.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:47:29.953 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-unexpected-clone' 2026-02-13T02:47:29.954 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:47:29.954 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:47:29.954 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:47:29.954 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:47:29.954 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:47:29.954 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.653483 2026-02-13T02:47:29.954 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.653483/$cluster-$name.asok' 2026-02-13T02:47:29.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.653483/$cluster-$name.asok' 2026-02-13T02:47:29.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:47:29.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:47:29.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:47:29.954 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-unexpected-clone/$name.log' 2026-02-13T02:47:29.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-unexpected-clone/$name.pid' 2026-02-13T02:47:29.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:47:29.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:47:29.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:47:29.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:47:29.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:47:29.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:47:29.955 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-unexpected-clone/0 2026-02-13T02:47:29.956 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:47:29.956 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=200a0260-9a03-4328-bf3f-ca485b731f65 2026-02-13T02:47:29.956 INFO:tasks.workunit.client.0.trial112.stdout:add osd0 200a0260-9a03-4328-bf3f-ca485b731f65 2026-02-13T02:47:29.957 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 200a0260-9a03-4328-bf3f-ca485b731f65' 2026-02-13T02:47:29.957 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:47:29.965 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDBkI5pb4taORAALXnBcNuIWRBhY/itrvRFKA== 2026-02-13T02:47:29.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDBkI5pb4taORAALXnBcNuIWRBhY/itrvRFKA=="}' 2026-02-13T02:47:29.966 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 200a0260-9a03-4328-bf3f-ca485b731f65 -i td/osd-unexpected-clone/0/new.json 2026-02-13T02:47:30.056 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:47:30.064 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-unexpected-clone/0/new.json 2026-02-13T02:47:30.066 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=61ed748e-0c83-461d-a19a-2a41771343bc --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/0 --osd-journal=td/osd-unexpected-clone/0/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.653483/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDBkI5pb4taORAALXnBcNuIWRBhY/itrvRFKA== --osd-uuid 200a0260-9a03-4328-bf3f-ca485b731f65 2026-02-13T02:47:30.081 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:30.075+0000 7f8bb67d2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:47:30.083 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:30.075+0000 7f8bb67d2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:47:30.083 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:30.075+0000 7f8bb67d2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:47:30.084 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:30.075+0000 7f8bb67d2a40 -1 bdev(0x564495396c00 td/osd-unexpected-clone/0/block) open stat got: (1) Operation not permitted 2026-02-13T02:47:30.084 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:30.075+0000 7f8bb67d2a40 -1 bluestore(td/osd-unexpected-clone/0) _read_fsid unparsable uuid 2026-02-13T02:47:30.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-unexpected-clone/0/keyring 2026-02-13T02:47:30.756 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:47:30.757 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-13T02:47:30.758 INFO:tasks.workunit.client.0.trial112.stdout:adding osd0 key to auth repository 2026-02-13T02:47:30.758 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-unexpected-clone/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:47:30.852 INFO:tasks.workunit.client.0.trial112.stdout:start osd.0 2026-02-13T02:47:30.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-13T02:47:30.853 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=61ed748e-0c83-461d-a19a-2a41771343bc --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/0 --osd-journal=td/osd-unexpected-clone/0/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.653483/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:47:30.854 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:47:30.854 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:47:30.854 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:47:30.867 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:30.859+0000 7f88ffddca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:47:30.868 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:30.859+0000 7f88ffddca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:47:30.869 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:30.863+0000 7f88ffddca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:47:30.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-13T02:47:30.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:47:30.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-13T02:47:30.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:47:30.948 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:47:30.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:47:30.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:47:30.949 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:47:30.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:47:30.949 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:47:31.041 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:47:31.455 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:31.447+0000 7f88ffddca40 -1 Falling back to public interface 2026-02-13T02:47:32.042 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:47:32.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:47:32.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:47:32.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:47:32.043 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:47:32.044 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:47:32.159 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:47:32.234 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:32.227+0000 7f88ffddca40 -1 osd.0 0 log_to_monitors true 2026-02-13T02:47:33.160 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:47:33.160 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:47:33.161 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:47:33.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:47:33.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:47:33.161 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-13T02:47:33.318 INFO:tasks.workunit.client.0.trial112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/388782722,v1:127.0.0.1:6801/388782722] [v2:127.0.0.1:6802/388782722,v1:127.0.0.1:6803/388782722] exists,up 200a0260-9a03-4328-bf3f-ca485b731f65 2026-02-13T02:47:33.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:47:33.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:47:33.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:47:33.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:45: TEST_recover_unexpected: run_osd td/osd-unexpected-clone 1 2026-02-13T02:47:33.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-unexpected-clone 2026-02-13T02:47:33.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:47:33.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-13T02:47:33.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:47:33.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-unexpected-clone/1 2026-02-13T02:47:33.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=61ed748e-0c83-461d-a19a-2a41771343bc --auth-supported=none --mon-host=127.0.0.1:7144 ' 2026-02-13T02:47:33.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:47:33.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:47:33.322 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:47:33.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-unexpected-clone/1' 2026-02-13T02:47:33.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-unexpected-clone/1/journal' 2026-02-13T02:47:33.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:47:33.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:47:33.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-unexpected-clone' 2026-02-13T02:47:33.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:47:33.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:47:33.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:47:33.323 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:47:33.323 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:47:33.323 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.653483 2026-02-13T02:47:33.323 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.653483/$cluster-$name.asok' 2026-02-13T02:47:33.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.653483/$cluster-$name.asok' 2026-02-13T02:47:33.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:47:33.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:47:33.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:47:33.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-unexpected-clone/$name.log' 2026-02-13T02:47:33.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-unexpected-clone/$name.pid' 2026-02-13T02:47:33.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:47:33.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:47:33.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:47:33.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:47:33.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:47:33.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:47:33.324 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-unexpected-clone/1 2026-02-13T02:47:33.325 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:47:33.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=118b7098-6eb3-42b1-86ba-3fb69ddd809e 2026-02-13T02:47:33.325 INFO:tasks.workunit.client.0.trial112.stdout:add osd1 118b7098-6eb3-42b1-86ba-3fb69ddd809e 2026-02-13T02:47:33.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 118b7098-6eb3-42b1-86ba-3fb69ddd809e' 2026-02-13T02:47:33.326 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:47:33.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDFkI5pUhi7ExAA+av5+zC+v864jwVj6DTfEA== 2026-02-13T02:47:33.334 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDFkI5pUhi7ExAA+av5+zC+v864jwVj6DTfEA=="}' 2026-02-13T02:47:33.335 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 118b7098-6eb3-42b1-86ba-3fb69ddd809e -i td/osd-unexpected-clone/1/new.json 2026-02-13T02:47:33.492 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:47:33.501 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-unexpected-clone/1/new.json 2026-02-13T02:47:33.502 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=61ed748e-0c83-461d-a19a-2a41771343bc --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/1 --osd-journal=td/osd-unexpected-clone/1/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.653483/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDFkI5pUhi7ExAA+av5+zC+v864jwVj6DTfEA== --osd-uuid 118b7098-6eb3-42b1-86ba-3fb69ddd809e 2026-02-13T02:47:33.526 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:33.519+0000 7f4304738a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:47:33.528 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:33.519+0000 7f4304738a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:47:33.529 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:33.519+0000 7f4304738a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:47:33.530 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:33.523+0000 7f4304738a40 -1 bdev(0x56158bc3ac00 td/osd-unexpected-clone/1/block) open stat got: (1) Operation not permitted 2026-02-13T02:47:33.530 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:33.523+0000 7f4304738a40 -1 bluestore(td/osd-unexpected-clone/1) _read_fsid unparsable uuid 2026-02-13T02:47:34.125 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-unexpected-clone/1/keyring 2026-02-13T02:47:34.125 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:47:34.126 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-13T02:47:34.126 INFO:tasks.workunit.client.0.trial112.stdout:adding osd1 key to auth repository 2026-02-13T02:47:34.127 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-unexpected-clone/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:47:34.339 INFO:tasks.workunit.client.0.trial112.stdout:start osd.1 2026-02-13T02:47:34.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-13T02:47:34.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=61ed748e-0c83-461d-a19a-2a41771343bc --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/1 --osd-journal=td/osd-unexpected-clone/1/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.653483/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:47:34.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:47:34.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:47:34.340 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:47:34.354 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:34.347+0000 7f1fd3784a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:47:34.356 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:34.347+0000 7f1fd3784a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:47:34.357 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:34.347+0000 7f1fd3784a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:47:34.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-13T02:47:34.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:47:34.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:47:34.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:47:34.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:47:34.504 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:47:34.505 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:47:34.505 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:47:34.505 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:47:34.506 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:47:34.670 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:47:34.678 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:34.671+0000 7f1fd3784a40 -1 Falling back to public interface 2026-02-13T02:47:35.298 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:35.291+0000 7f1fd3784a40 -1 osd.1 0 log_to_monitors true 2026-02-13T02:47:35.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:47:35.671 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:47:35.672 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:47:35.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:47:35.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:47:35.672 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:47:35.833 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:47:36.834 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:47:36.835 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:47:36.835 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:47:36.836 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:47:36.836 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:47:36.836 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:47:36.993 INFO:tasks.workunit.client.0.trial112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3883523381,v1:127.0.0.1:6811/3883523381] [v2:127.0.0.1:6812/3883523381,v1:127.0.0.1:6813/3883523381] exists,up 118b7098-6eb3-42b1-86ba-3fb69ddd809e 2026-02-13T02:47:36.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:47:36.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:47:36.996 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:47:36.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:46: TEST_recover_unexpected: run_osd td/osd-unexpected-clone 2 2026-02-13T02:47:36.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-unexpected-clone 2026-02-13T02:47:36.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-13T02:47:36.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-13T02:47:36.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-13T02:47:36.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-unexpected-clone/2 2026-02-13T02:47:36.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=61ed748e-0c83-461d-a19a-2a41771343bc --auth-supported=none --mon-host=127.0.0.1:7144 ' 2026-02-13T02:47:36.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:47:36.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:47:36.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:47:36.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-unexpected-clone/2' 2026-02-13T02:47:36.997 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-unexpected-clone/2/journal' 2026-02-13T02:47:36.998 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-13T02:47:36.998 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-13T02:47:36.998 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-unexpected-clone' 2026-02-13T02:47:36.998 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-13T02:47:36.998 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:47:36.998 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:47:36.998 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:47:36.998 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:47:36.998 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.653483 2026-02-13T02:47:36.998 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.653483/$cluster-$name.asok' 2026-02-13T02:47:36.998 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.653483/$cluster-$name.asok' 2026-02-13T02:47:36.998 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:47:36.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-13T02:47:36.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-13T02:47:36.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-unexpected-clone/$name.log' 2026-02-13T02:47:36.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-unexpected-clone/$name.pid' 2026-02-13T02:47:36.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:47:36.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:47:36.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:47:36.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:47:36.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-13T02:47:36.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-13T02:47:36.999 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-unexpected-clone/2 2026-02-13T02:47:36.999 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-13T02:47:37.000 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=06345ee9-d0aa-4b2e-8a96-42eaf7210230 2026-02-13T02:47:37.000 INFO:tasks.workunit.client.0.trial112.stdout:add osd2 06345ee9-d0aa-4b2e-8a96-42eaf7210230 2026-02-13T02:47:37.001 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 06345ee9-d0aa-4b2e-8a96-42eaf7210230' 2026-02-13T02:47:37.001 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-13T02:47:37.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDJkI5pcBFvABAABF784PljuJ3ZeDHco4VXbg== 2026-02-13T02:47:37.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDJkI5pcBFvABAABF784PljuJ3ZeDHco4VXbg=="}' 2026-02-13T02:47:37.011 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 06345ee9-d0aa-4b2e-8a96-42eaf7210230 -i td/osd-unexpected-clone/2/new.json 2026-02-13T02:47:37.160 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:47:37.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-unexpected-clone/2/new.json 2026-02-13T02:47:37.170 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=61ed748e-0c83-461d-a19a-2a41771343bc --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/2 --osd-journal=td/osd-unexpected-clone/2/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.653483/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDJkI5pcBFvABAABF784PljuJ3ZeDHco4VXbg== --osd-uuid 06345ee9-d0aa-4b2e-8a96-42eaf7210230 2026-02-13T02:47:37.184 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:37.175+0000 7fb452b44a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:47:37.186 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:37.179+0000 7fb452b44a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:47:37.187 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:37.179+0000 7fb452b44a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:47:37.188 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:37.179+0000 7fb452b44a40 -1 bdev(0x55637309cc00 td/osd-unexpected-clone/2/block) open stat got: (1) Operation not permitted 2026-02-13T02:47:37.188 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:37.179+0000 7fb452b44a40 -1 bluestore(td/osd-unexpected-clone/2) _read_fsid unparsable uuid 2026-02-13T02:47:37.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-unexpected-clone/2/keyring 2026-02-13T02:47:37.809 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-13T02:47:37.810 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-13T02:47:37.810 INFO:tasks.workunit.client.0.trial112.stdout:adding osd2 key to auth repository 2026-02-13T02:47:37.811 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-unexpected-clone/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-13T02:47:38.008 INFO:tasks.workunit.client.0.trial112.stdout:start osd.2 2026-02-13T02:47:38.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-13T02:47:38.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=61ed748e-0c83-461d-a19a-2a41771343bc --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/2 --osd-journal=td/osd-unexpected-clone/2/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.653483/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:47:38.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-13T02:47:38.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-13T02:47:38.009 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-13T02:47:38.031 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:38.023+0000 7fc8ba3b5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:47:38.033 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:38.023+0000 7fc8ba3b5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:47:38.033 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:38.027+0000 7fc8ba3b5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:47:38.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-13T02:47:38.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:47:38.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T02:47:38.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:47:38.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:47:38.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:47:38.169 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:47:38.170 INFO:tasks.workunit.client.0.trial112.stdout:0 2026-02-13T02:47:38.170 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:47:38.170 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:47:38.325 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:47:38.447 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:38.439+0000 7fc8ba3b5a40 -1 Falling back to public interface 2026-02-13T02:47:39.088 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:39.079+0000 7fc8ba3b5a40 -1 osd.2 0 log_to_monitors true 2026-02-13T02:47:39.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:47:39.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:47:39.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:47:39.327 INFO:tasks.workunit.client.0.trial112.stdout:1 2026-02-13T02:47:39.327 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:47:39.328 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:47:39.480 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:47:40.482 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:47:40.483 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:47:40.483 INFO:tasks.workunit.client.0.trial112.stdout:2 2026-02-13T02:47:40.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-13T02:47:40.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:47:40.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:47:40.638 INFO:tasks.workunit.client.0.trial112.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2473178630,v1:127.0.0.1:6819/2473178630] [v2:127.0.0.1:6820/2473178630,v1:127.0.0.1:6821/2473178630] exists,up 06345ee9-d0aa-4b2e-8a96-42eaf7210230 2026-02-13T02:47:40.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:47:40.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:47:40.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:47:40.638 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:48: TEST_recover_unexpected: ceph osd pool create foo 1 2026-02-13T02:47:40.842 INFO:tasks.workunit.client.0.trial112.stderr:pool 'foo' created 2026-02-13T02:47:40.852 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:49: TEST_recover_unexpected: rados -p foo put foo /etc/passwd 2026-02-13T02:47:41.336 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:50: TEST_recover_unexpected: rados -p foo mksnap snap 2026-02-13T02:47:41.381 INFO:tasks.workunit.client.0.trial112.stdout:created pool foo snap snap 2026-02-13T02:47:41.383 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:51: TEST_recover_unexpected: rados -p foo put foo /etc/group 2026-02-13T02:47:41.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:53: TEST_recover_unexpected: wait_for_clean 2026-02-13T02:47:41.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:47:41.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:47:41.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:47:41.409 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:47:41.409 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:47:41.409 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:47:41.409 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:47:41.409 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:47:41.409 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:47:41.507 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:47:41.508 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:47:41.508 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:47:41.508 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:47:41.508 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:47:41.508 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:47:41.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:47:41.669 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:47:41.669 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:47:41.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:47:41.669 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:47:41.669 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:47:41.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-13T02:47:41.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-13T02:47:41.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-13T02:47:41.743 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:47:41.744 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:47:41.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672963 2026-02-13T02:47:41.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672963 2026-02-13T02:47:41.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-42949672963' 2026-02-13T02:47:41.813 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:47:41.813 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:47:41.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509443 2026-02-13T02:47:41.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509443 2026-02-13T02:47:41.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-42949672963 2-64424509443' 2026-02-13T02:47:41.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:47:41.888 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-13T02:47:41.888 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:47:41.889 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:47:41.890 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-13T02:47:41.890 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:47:41.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-13T02:47:41.891 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-13T02:47:41.892 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.0 seq 21474836483 2026-02-13T02:47:41.892 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:47:42.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-13T02:47:42.059 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:47:43.060 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:47:43.061 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:47:43.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-13T02:47:43.221 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:47:43.222 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672963 2026-02-13T02:47:43.222 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:47:43.223 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:47:43.224 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672963 2026-02-13T02:47:43.224 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:47:43.225 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672963 2026-02-13T02:47:43.225 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.1 seq 42949672963 2026-02-13T02:47:43.226 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2026-02-13T02:47:43.226 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:47:43.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672963 -lt 42949672963 2026-02-13T02:47:43.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:47:43.391 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509443 2026-02-13T02:47:43.391 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:47:43.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:47:43.393 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509443 2026-02-13T02:47:43.393 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:47:43.395 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509443 2026-02-13T02:47:43.395 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509443' 2026-02-13T02:47:43.395 INFO:tasks.workunit.client.0.trial112.stdout:waiting osd.2 seq 64424509443 2026-02-13T02:47:43.395 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:47:43.569 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509443 -lt 64424509443 2026-02-13T02:47:43.570 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:47:43.570 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:47:43.570 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:47:43.766 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:47:43.767 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:47:43.767 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:47:43.767 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:47:43.767 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:47:43.767 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:47:43.767 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:47:43.767 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:47:43.928 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:47:43.928 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:47:43.928 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:47:43.929 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:47:44.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:47:44.119 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:47:44.120 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:47:44.121 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:55: TEST_recover_unexpected: get_primary foo foo 2026-02-13T02:47:44.121 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=foo 2026-02-13T02:47:44.121 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=foo 2026-02-13T02:47:44.121 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map foo foo 2026-02-13T02:47:44.121 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-13T02:47:44.275 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:55: TEST_recover_unexpected: local osd=1 2026-02-13T02:47:44.277 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:57: TEST_recover_unexpected: objectstore_tool td/osd-unexpected-clone 1 --op list foo 2026-02-13T02:47:44.277 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:57: TEST_recover_unexpected: grep snapid.:1 2026-02-13T02:47:44.277 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-unexpected-clone 2026-02-13T02:47:44.277 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:47:44.277 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:47:44.277 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:47:44.277 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-unexpected-clone 1 --op list foo 2026-02-13T02:47:44.278 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-unexpected-clone 2026-02-13T02:47:44.278 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:47:44.278 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:47:44.278 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:47:44.278 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-unexpected-clone TERM osd.1 2026-02-13T02:47:44.278 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:47:44.278 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:47:44.278 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:47:44.278 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:47:44.278 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:47:44.584 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:47:44.584 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-unexpected-clone 1 --op list foo 2026-02-13T02:47:44.585 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-unexpected-clone 2026-02-13T02:47:44.585 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:47:44.585 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:47:44.585 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:47:44.585 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-unexpected-clone/1 2026-02-13T02:47:44.585 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-unexpected-clone/1 --op list foo 2026-02-13T02:47:44.756 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-unexpected-clone 1 2026-02-13T02:47:44.756 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-unexpected-clone 2026-02-13T02:47:44.756 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:47:44.756 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:47:44.756 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:47:44.756 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-unexpected-clone/1 2026-02-13T02:47:44.756 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=61ed748e-0c83-461d-a19a-2a41771343bc --auth-supported=none --mon-host=127.0.0.1:7144 ' 2026-02-13T02:47:44.757 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:47:44.757 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:47:44.757 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:47:44.757 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-unexpected-clone/1' 2026-02-13T02:47:44.757 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-unexpected-clone/1/journal' 2026-02-13T02:47:44.757 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:47:44.757 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:47:44.757 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-unexpected-clone' 2026-02-13T02:47:44.757 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:47:44.757 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:47:44.757 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:47:44.757 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:47:44.758 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:47:44.758 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.653483 2026-02-13T02:47:44.758 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.653483/$cluster-$name.asok' 2026-02-13T02:47:44.758 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.653483/$cluster-$name.asok' 2026-02-13T02:47:44.758 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:47:44.758 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-unexpected-clone/$name.log' 2026-02-13T02:47:44.758 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-unexpected-clone/$name.pid' 2026-02-13T02:47:44.758 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:47:44.758 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:47:44.758 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:47:44.758 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:47:44.758 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:47:44.759 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:47:44.759 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-unexpected-clone/1 2026-02-13T02:47:44.760 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:47:44.760 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:47:44.760 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=61ed748e-0c83-461d-a19a-2a41771343bc --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/1 --osd-journal=td/osd-unexpected-clone/1/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.653483/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:47:44.760 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-unexpected-clone/1/whoami 2026-02-13T02:47:44.760 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:47:44.760 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:47:44.760 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:47:44.760 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:47:44.773 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:44.766+0000 7fe5d1609a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:47:44.775 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:44.766+0000 7fe5d1609a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:47:44.776 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:44.766+0000 7fe5d1609a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:47:44.920 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:47:44.920 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:47:44.920 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:47:44.921 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:47:44.921 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:47:44.921 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:47:44.921 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:47:44.921 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:47:44.921 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:47:44.921 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:47:45.082 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:47:45.131 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:45.122+0000 7fe5d1609a40 -1 Falling back to public interface 2026-02-13T02:47:45.740 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:45.730+0000 7fe5d1609a40 -1 osd.1 18 log_to_monitors true 2026-02-13T02:47:46.084 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:47:46.084 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:47:46.084 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:47:46.084 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:47:46.085 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:47:46.085 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:47:46.248 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 22 up_thru 22 down_at 19 last_clean_interval [10,18) [v2:127.0.0.1:6810/2208404611,v1:127.0.0.1:6811/2208404611] [v2:127.0.0.1:6812/2208404611,v1:127.0.0.1:6813/2208404611] exists,up 118b7098-6eb3-42b1-86ba-3fb69ddd809e 2026-02-13T02:47:46.249 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:47:46.249 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:47:46.249 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:47:46.249 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:47:46.249 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:47:46.249 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:47:46.249 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:47:46.249 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:47:46.249 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:47:46.249 INFO:tasks.workunit.client.0.trial112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:47:46.250 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:47:46.250 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:47:46.250 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:47:46.343 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:47:46.343 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:47:46.343 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:47:46.343 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:47:46.343 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:47:46.344 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:47:46.509 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:47:46.509 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:47:46.510 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:47:46.510 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:47:46.510 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:47:46.510 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:47:46.580 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836485 2026-02-13T02:47:46.581 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836485 2026-02-13T02:47:46.581 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485' 2026-02-13T02:47:46.581 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:47:46.581 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:47:46.652 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280514 2026-02-13T02:47:46.652 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280514 2026-02-13T02:47:46.653 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-94489280514' 2026-02-13T02:47:46.653 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:47:46.653 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:47:46.725 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509445 2026-02-13T02:47:46.725 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509445 2026-02-13T02:47:46.725 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-94489280514 2-64424509445' 2026-02-13T02:47:46.725 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:47:46.725 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836485 2026-02-13T02:47:46.725 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:47:46.726 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:47:46.727 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836485 2026-02-13T02:47:46.727 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:47:46.728 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836485 2026-02-13T02:47:46.728 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-13T02:47:46.728 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 21474836485 2026-02-13T02:47:46.728 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:47:46.894 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836485 2026-02-13T02:47:46.894 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:47:47.895 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:47:47.895 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:47:48.055 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836485 2026-02-13T02:47:48.055 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:47:49.056 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:47:49.057 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:47:49.217 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 21474836485 2026-02-13T02:47:49.218 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:47:49.218 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-94489280514 2026-02-13T02:47:49.218 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:47:49.219 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:47:49.220 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-94489280514 2026-02-13T02:47:49.220 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:47:49.221 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280514 2026-02-13T02:47:49.221 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 94489280514' 2026-02-13T02:47:49.221 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 94489280514 2026-02-13T02:47:49.222 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:47:49.383 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280514 -lt 94489280514 2026-02-13T02:47:49.383 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:47:49.384 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509445 2026-02-13T02:47:49.384 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:47:49.385 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:47:49.386 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509445 2026-02-13T02:47:49.386 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:47:49.387 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509445 2026-02-13T02:47:49.388 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2026-02-13T02:47:49.388 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 64424509445 2026-02-13T02:47:49.388 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:47:49.556 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509445 -lt 64424509445 2026-02-13T02:47:49.557 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:47:49.557 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:47:49.557 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:47:49.749 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:47:49.749 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:47:49.749 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:47:49.750 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:47:49.750 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:47:49.750 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:47:49.750 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:47:49.750 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:47:49.903 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:47:49.904 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:47:49.904 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:47:49.904 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:47:50.094 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:47:50.094 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:47:50.094 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:47:50.094 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:57: TEST_recover_unexpected: JSON='["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:47:50.095 INFO:tasks.workunit.client.0.trial112.stdout:JSON is ["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}] 2026-02-13T02:47:50.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:58: TEST_recover_unexpected: echo 'JSON is ["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-13T02:47:50.095 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:59: TEST_recover_unexpected: rm -f td/osd-unexpected-clone/_ td/osd-unexpected-clone/data 2026-02-13T02:47:50.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:60: TEST_recover_unexpected: objectstore_tool td/osd-unexpected-clone 1 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' get-attr _ 2026-02-13T02:47:50.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-unexpected-clone 2026-02-13T02:47:50.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:47:50.097 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-13T02:47:50.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:47:50.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-unexpected-clone 1 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' get-attr _ 2026-02-13T02:47:50.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-unexpected-clone 2026-02-13T02:47:50.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:47:50.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-13T02:47:50.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:47:50.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-unexpected-clone TERM osd.1 2026-02-13T02:47:50.098 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:47:50.098 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:47:50.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:47:50.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:47:50.098 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:47:50.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:47:50.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-unexpected-clone 1 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' get-attr _ 2026-02-13T02:47:50.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-unexpected-clone 2026-02-13T02:47:50.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:47:50.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-13T02:47:50.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:47:50.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-unexpected-clone/1 2026-02-13T02:47:50.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-unexpected-clone/1 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' get-attr _ 2026-02-13T02:47:50.563 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-unexpected-clone 1 2026-02-13T02:47:50.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-unexpected-clone 2026-02-13T02:47:50.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:47:50.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-13T02:47:50.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:47:50.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-unexpected-clone/1 2026-02-13T02:47:50.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=61ed748e-0c83-461d-a19a-2a41771343bc --auth-supported=none --mon-host=127.0.0.1:7144 ' 2026-02-13T02:47:50.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:47:50.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:47:50.564 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:47:50.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-unexpected-clone/1' 2026-02-13T02:47:50.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-unexpected-clone/1/journal' 2026-02-13T02:47:50.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:47:50.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:47:50.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-unexpected-clone' 2026-02-13T02:47:50.565 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:47:50.565 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:47:50.565 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:47:50.565 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:47:50.565 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:47:50.565 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.653483 2026-02-13T02:47:50.565 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.653483/$cluster-$name.asok' 2026-02-13T02:47:50.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.653483/$cluster-$name.asok' 2026-02-13T02:47:50.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:47:50.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-unexpected-clone/$name.log' 2026-02-13T02:47:50.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-unexpected-clone/$name.pid' 2026-02-13T02:47:50.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:47:50.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:47:50.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:47:50.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:47:50.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:47:50.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:47:50.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-unexpected-clone/1 2026-02-13T02:47:50.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-13T02:47:50.567 INFO:tasks.workunit.client.0.trial112.stderr:start osd.1 2026-02-13T02:47:50.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=61ed748e-0c83-461d-a19a-2a41771343bc --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/1 --osd-journal=td/osd-unexpected-clone/1/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.653483/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:47:50.568 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-unexpected-clone/1/whoami 2026-02-13T02:47:50.568 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-13T02:47:50.568 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:47:50.568 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:47:50.568 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:47:50.579 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:50.574+0000 7f33f9fc0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:47:50.581 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:50.574+0000 7f33f9fc0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:47:50.582 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:50.578+0000 7f33f9fc0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:47:50.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-13T02:47:50.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:47:50.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-13T02:47:50.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:47:50.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:47:50.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:47:50.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:47:50.724 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:47:50.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:47:50.724 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:47:50.884 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:47:50.979 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:50.974+0000 7f33f9fc0a40 -1 Falling back to public interface 2026-02-13T02:47:51.601 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:51.594+0000 7f33f9fc0a40 -1 osd.1 23 log_to_monitors true 2026-02-13T02:47:51.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:47:51.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:47:51.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:47:51.887 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:47:51.887 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:47:51.888 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-13T02:47:52.045 INFO:tasks.workunit.client.0.trial112.stderr:osd.1 up in weight 1 up_from 27 up_thru 27 down_at 24 last_clean_interval [22,23) [v2:127.0.0.1:6810/2304447455,v1:127.0.0.1:6811/2304447455] [v2:127.0.0.1:6812/2304447455,v1:127.0.0.1:6813/2304447455] exists,up 118b7098-6eb3-42b1-86ba-3fb69ddd809e 2026-02-13T02:47:52.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:47:52.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:47:52.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:47:52.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:47:52.045 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:47:52.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:47:52.046 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:47:52.046 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:47:52.046 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:47:52.046 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:47:52.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:47:52.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:47:52.047 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:47:52.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:47:52.141 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:47:52.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:47:52.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:47:52.142 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:47:52.142 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:47:52.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:47:52.313 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:47:52.313 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:47:52.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:47:52.313 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:47:52.313 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:47:52.383 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836487 2026-02-13T02:47:52.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836487 2026-02-13T02:47:52.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487' 2026-02-13T02:47:52.384 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:47:52.384 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:47:52.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964116994 2026-02-13T02:47:52.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964116994 2026-02-13T02:47:52.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487 1-115964116994' 2026-02-13T02:47:52.457 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:47:52.457 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:47:52.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509447 2026-02-13T02:47:52.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509447 2026-02-13T02:47:52.528 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487 1-115964116994 2-64424509447' 2026-02-13T02:47:52.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:47:52.529 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836487 2026-02-13T02:47:52.529 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:47:52.529 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:47:52.530 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836487 2026-02-13T02:47:52.530 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:47:52.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836487 2026-02-13T02:47:52.531 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2026-02-13T02:47:52.532 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 21474836487 2026-02-13T02:47:52.532 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:47:52.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836487 2026-02-13T02:47:52.702 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:47:53.703 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:47:53.704 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:47:53.864 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836487 2026-02-13T02:47:53.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:47:54.865 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:47:54.866 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:47:55.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836487 2026-02-13T02:47:55.032 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:47:55.033 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-115964116994 2026-02-13T02:47:55.033 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:47:55.033 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:47:55.034 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-115964116994 2026-02-13T02:47:55.034 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:47:55.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=115964116994 2026-02-13T02:47:55.036 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 115964116994' 2026-02-13T02:47:55.036 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 115964116994 2026-02-13T02:47:55.036 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:47:55.199 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 115964116994 -lt 115964116994 2026-02-13T02:47:55.199 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:47:55.200 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509447 2026-02-13T02:47:55.200 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:47:55.201 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:47:55.202 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509447 2026-02-13T02:47:55.202 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:47:55.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509447 2026-02-13T02:47:55.204 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2026-02-13T02:47:55.204 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 64424509447 2026-02-13T02:47:55.204 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:47:55.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509447 -lt 64424509447 2026-02-13T02:47:55.368 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:47:55.368 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:47:55.368 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:47:55.565 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:47:55.566 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:47:55.566 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:47:55.566 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:47:55.566 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:47:55.566 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:47:55.566 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:47:55.566 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:47:55.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:47:55.731 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:47:55.732 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:47:55.732 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:47:55.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:47:55.933 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:47:55.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:47:55.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:61: TEST_recover_unexpected: objectstore_tool td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' get-bytes td/osd-unexpected-clone/data 2026-02-13T02:47:55.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-unexpected-clone 2026-02-13T02:47:55.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:47:55.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-13T02:47:55.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:47:55.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' get-bytes td/osd-unexpected-clone/data 2026-02-13T02:47:55.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-unexpected-clone 2026-02-13T02:47:55.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:47:55.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-13T02:47:55.934 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:47:55.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-unexpected-clone TERM osd.2 2026-02-13T02:47:55.935 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:47:55.935 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:47:55.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:47:55.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:47:55.935 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:47:56.240 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:47:56.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' get-bytes td/osd-unexpected-clone/data 2026-02-13T02:47:56.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-unexpected-clone 2026-02-13T02:47:56.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:47:56.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-13T02:47:56.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:47:56.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-unexpected-clone/2 2026-02-13T02:47:56.241 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-unexpected-clone/2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' get-bytes td/osd-unexpected-clone/data 2026-02-13T02:47:56.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-unexpected-clone 2 2026-02-13T02:47:56.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-unexpected-clone 2026-02-13T02:47:56.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:47:56.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-13T02:47:56.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:47:56.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-unexpected-clone/2 2026-02-13T02:47:56.388 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=61ed748e-0c83-461d-a19a-2a41771343bc --auth-supported=none --mon-host=127.0.0.1:7144 ' 2026-02-13T02:47:56.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:47:56.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:47:56.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:47:56.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-unexpected-clone/2' 2026-02-13T02:47:56.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-unexpected-clone/2/journal' 2026-02-13T02:47:56.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:47:56.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:47:56.389 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-unexpected-clone' 2026-02-13T02:47:56.389 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:47:56.389 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:47:56.389 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:47:56.389 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:47:56.390 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:47:56.390 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.653483 2026-02-13T02:47:56.390 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.653483/$cluster-$name.asok' 2026-02-13T02:47:56.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.653483/$cluster-$name.asok' 2026-02-13T02:47:56.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:47:56.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-unexpected-clone/$name.log' 2026-02-13T02:47:56.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-unexpected-clone/$name.pid' 2026-02-13T02:47:56.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:47:56.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:47:56.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:47:56.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:47:56.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:47:56.390 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:47:56.391 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-unexpected-clone/2 2026-02-13T02:47:56.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-13T02:47:56.392 INFO:tasks.workunit.client.0.trial112.stderr:start osd.2 2026-02-13T02:47:56.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=61ed748e-0c83-461d-a19a-2a41771343bc --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/2 --osd-journal=td/osd-unexpected-clone/2/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.653483/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:47:56.392 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-unexpected-clone/2/whoami 2026-02-13T02:47:56.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-13T02:47:56.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:47:56.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:47:56.392 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:47:56.411 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:56.406+0000 7f0ac3fbfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:47:56.412 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:56.406+0000 7f0ac3fbfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:47:56.413 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:56.410+0000 7f0ac3fbfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:47:56.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-13T02:47:56.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:47:56.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T02:47:56.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:47:56.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:47:56.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:47:56.552 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:47:56.552 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:47:56.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:47:56.553 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:47:56.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:47:56.863 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:56.858+0000 7f0ac3fbfa40 -1 Falling back to public interface 2026-02-13T02:47:57.542 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:47:57.538+0000 7f0ac3fbfa40 -1 osd.2 29 log_to_monitors true 2026-02-13T02:47:57.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:47:57.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:47:57.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:47:57.710 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:47:57.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:47:57.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:47:57.868 INFO:tasks.workunit.client.0.trial112.stderr:osd.2 up in weight 1 up_from 32 up_thru 0 down_at 29 last_clean_interval [15,29) [v2:127.0.0.1:6818/4141650305,v1:127.0.0.1:6819/4141650305] [v2:127.0.0.1:6820/4141650305,v1:127.0.0.1:6821/4141650305] exists,up 06345ee9-d0aa-4b2e-8a96-42eaf7210230 2026-02-13T02:47:57.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:47:57.868 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:47:57.869 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:47:57.869 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:47:57.869 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:47:57.869 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:47:57.869 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:47:57.869 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:47:57.870 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:47:57.870 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:47:57.870 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:47:57.870 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:47:57.870 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:47:57.968 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:47:57.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:47:57.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:47:57.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:47:57.969 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:47:57.969 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:47:58.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:47:58.137 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:47:58.137 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:47:58.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:47:58.137 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:47:58.137 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:47:58.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836489 2026-02-13T02:47:58.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836489 2026-02-13T02:47:58.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489' 2026-02-13T02:47:58.211 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:47:58.211 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:47:58.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964116996 2026-02-13T02:47:58.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964116996 2026-02-13T02:47:58.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-115964116996' 2026-02-13T02:47:58.284 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:47:58.285 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:47:58.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953474 2026-02-13T02:47:58.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953474 2026-02-13T02:47:58.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-115964116996 2-137438953474' 2026-02-13T02:47:58.356 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:47:58.357 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836489 2026-02-13T02:47:58.357 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:47:58.358 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:47:58.359 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836489 2026-02-13T02:47:58.359 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:47:58.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836489 2026-02-13T02:47:58.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2026-02-13T02:47:58.360 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 21474836489 2026-02-13T02:47:58.361 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:47:58.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836489 2026-02-13T02:47:58.524 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:47:59.525 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:47:59.526 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:47:59.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836489 2026-02-13T02:47:59.689 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:48:00.691 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-13T02:48:00.692 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:48:00.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836490 -lt 21474836489 2026-02-13T02:48:00.855 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:48:00.855 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-115964116996 2026-02-13T02:48:00.856 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:48:00.856 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:48:00.857 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-115964116996 2026-02-13T02:48:00.857 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:48:00.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=115964116996 2026-02-13T02:48:00.858 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 115964116996' 2026-02-13T02:48:00.858 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 115964116996 2026-02-13T02:48:00.859 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:48:01.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 115964116996 -lt 115964116996 2026-02-13T02:48:01.020 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:48:01.020 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-137438953474 2026-02-13T02:48:01.021 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:48:01.021 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:48:01.022 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-137438953474 2026-02-13T02:48:01.022 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:48:01.024 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953474 2026-02-13T02:48:01.024 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 137438953474' 2026-02-13T02:48:01.024 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 137438953474 2026-02-13T02:48:01.024 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:48:01.196 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953474 -lt 137438953474 2026-02-13T02:48:01.197 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:48:01.197 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:48:01.197 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:48:01.394 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:48:01.394 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:48:01.395 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:48:01.395 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:48:01.395 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:48:01.395 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:48:01.396 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:48:01.396 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:48:01.555 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:48:01.556 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:48:01.556 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:48:01.556 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:48:01.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:48:01.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:48:01.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:48:01.754 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:63: TEST_recover_unexpected: rados -p foo rmsnap snap 2026-02-13T02:48:01.821 INFO:tasks.workunit.client.0.trial112.stdout:removed pool foo snap snap 2026-02-13T02:48:01.823 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:65: TEST_recover_unexpected: sleep 5 2026-02-13T02:48:06.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:67: TEST_recover_unexpected: objectstore_tool td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes td/osd-unexpected-clone/data 2026-02-13T02:48:06.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-unexpected-clone 2026-02-13T02:48:06.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:48:06.824 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-13T02:48:06.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:48:06.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes td/osd-unexpected-clone/data 2026-02-13T02:48:06.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-unexpected-clone 2026-02-13T02:48:06.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:48:06.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-13T02:48:06.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:48:06.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-unexpected-clone TERM osd.2 2026-02-13T02:48:06.825 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:48:06.825 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:48:06.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:48:06.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:48:06.825 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:48:07.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:48:07.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes td/osd-unexpected-clone/data 2026-02-13T02:48:07.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-unexpected-clone 2026-02-13T02:48:07.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:48:07.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-13T02:48:07.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:48:07.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-unexpected-clone/2 2026-02-13T02:48:07.135 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-unexpected-clone/2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes td/osd-unexpected-clone/data 2026-02-13T02:48:07.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-unexpected-clone 2 2026-02-13T02:48:07.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-unexpected-clone 2026-02-13T02:48:07.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:48:07.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-13T02:48:07.406 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:48:07.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-unexpected-clone/2 2026-02-13T02:48:07.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=61ed748e-0c83-461d-a19a-2a41771343bc --auth-supported=none --mon-host=127.0.0.1:7144 ' 2026-02-13T02:48:07.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:48:07.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:48:07.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:48:07.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-unexpected-clone/2' 2026-02-13T02:48:07.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-unexpected-clone/2/journal' 2026-02-13T02:48:07.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:48:07.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:48:07.407 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-unexpected-clone' 2026-02-13T02:48:07.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:48:07.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:48:07.407 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:48:07.407 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:48:07.407 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:48:07.407 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.653483 2026-02-13T02:48:07.408 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.653483/$cluster-$name.asok' 2026-02-13T02:48:07.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.653483/$cluster-$name.asok' 2026-02-13T02:48:07.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:48:07.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-unexpected-clone/$name.log' 2026-02-13T02:48:07.408 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-unexpected-clone/$name.pid' 2026-02-13T02:48:07.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:48:07.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:48:07.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:48:07.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:48:07.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:48:07.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:48:07.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-unexpected-clone/2 2026-02-13T02:48:07.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-13T02:48:07.409 INFO:tasks.workunit.client.0.trial112.stderr:start osd.2 2026-02-13T02:48:07.409 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=61ed748e-0c83-461d-a19a-2a41771343bc --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/2 --osd-journal=td/osd-unexpected-clone/2/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.653483/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:48:07.410 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-unexpected-clone/2/whoami 2026-02-13T02:48:07.410 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-13T02:48:07.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:48:07.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:48:07.411 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:48:07.428 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:48:07.422+0000 7fbceafa6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:48:07.429 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:48:07.426+0000 7fbceafa6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:48:07.430 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:48:07.426+0000 7fbceafa6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:48:07.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-13T02:48:07.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:48:07.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T02:48:07.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:48:07.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:48:07.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:48:07.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:48:07.567 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:48:07.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:48:07.567 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:48:07.731 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:48:07.814 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:48:07.810+0000 7fbceafa6a40 -1 Falling back to public interface 2026-02-13T02:48:08.488 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:48:08.482+0000 7fbceafa6a40 -1 osd.2 35 log_to_monitors true 2026-02-13T02:48:08.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:48:08.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:48:08.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:48:08.733 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:48:08.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:48:08.733 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:48:08.895 INFO:tasks.workunit.client.0.trial112.stderr:osd.2 up in weight 1 up_from 39 up_thru 0 down_at 36 last_clean_interval [32,35) [v2:127.0.0.1:6818/3419801980,v1:127.0.0.1:6819/3419801980] [v2:127.0.0.1:6820/3419801980,v1:127.0.0.1:6821/3419801980] exists,up 06345ee9-d0aa-4b2e-8a96-42eaf7210230 2026-02-13T02:48:08.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:48:08.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:48:08.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:48:08.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:48:08.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:48:08.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:48:08.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:48:08.896 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:48:08.897 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:48:08.897 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:48:08.897 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:48:08.897 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:48:08.897 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:48:08.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:48:08.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:48:08.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:48:08.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:48:08.991 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:48:08.992 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:48:09.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:48:09.151 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:48:09.151 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:48:09.151 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:48:09.152 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:48:09.152 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:48:09.224 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836493 2026-02-13T02:48:09.224 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836493 2026-02-13T02:48:09.224 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493' 2026-02-13T02:48:09.224 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:48:09.225 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:48:09.293 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964116999 2026-02-13T02:48:09.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964116999 2026-02-13T02:48:09.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-115964116999' 2026-02-13T02:48:09.294 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:48:09.294 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:48:09.366 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=167503724546 2026-02-13T02:48:09.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 167503724546 2026-02-13T02:48:09.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-115964116999 2-167503724546' 2026-02-13T02:48:09.367 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:48:09.367 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836493 2026-02-13T02:48:09.367 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:48:09.368 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:48:09.369 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836493 2026-02-13T02:48:09.369 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:48:09.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836493 2026-02-13T02:48:09.370 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2026-02-13T02:48:09.370 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 21474836493 2026-02-13T02:48:09.371 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:48:09.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836493 2026-02-13T02:48:09.541 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-13T02:48:10.542 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-13T02:48:10.543 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:48:10.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836493 -lt 21474836493 2026-02-13T02:48:10.708 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:48:10.709 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-115964116999 2026-02-13T02:48:10.709 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:48:10.710 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:48:10.711 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-115964116999 2026-02-13T02:48:10.711 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:48:10.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=115964116999 2026-02-13T02:48:10.712 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 115964116999' 2026-02-13T02:48:10.712 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 115964116999 2026-02-13T02:48:10.713 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:48:10.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 115964116999 -lt 115964116999 2026-02-13T02:48:10.881 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:48:10.882 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-167503724546 2026-02-13T02:48:10.882 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:48:10.883 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:48:10.883 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-167503724546 2026-02-13T02:48:10.884 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:48:10.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=167503724546 2026-02-13T02:48:10.885 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 167503724546' 2026-02-13T02:48:10.885 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 167503724546 2026-02-13T02:48:10.885 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:48:11.055 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 167503724546 -lt 167503724546 2026-02-13T02:48:11.055 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:48:11.056 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:48:11.056 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:48:11.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:48:11.246 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:48:11.247 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:48:11.247 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:48:11.247 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:48:11.247 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:48:11.247 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:48:11.247 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:48:11.403 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:48:11.404 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:48:11.404 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:48:11.404 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:48:11.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:48:11.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:48:11.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:48:11.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:68: TEST_recover_unexpected: objectstore_tool td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' set-attr _ td/osd-unexpected-clone/_ 2026-02-13T02:48:11.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-unexpected-clone 2026-02-13T02:48:11.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-13T02:48:11.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-13T02:48:11.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-13T02:48:11.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' set-attr _ td/osd-unexpected-clone/_ 2026-02-13T02:48:11.592 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-unexpected-clone 2026-02-13T02:48:11.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-13T02:48:11.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-13T02:48:11.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-13T02:48:11.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-unexpected-clone TERM osd.2 2026-02-13T02:48:11.593 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:48:11.593 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:48:11.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:48:11.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:48:11.593 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:48:11.901 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:48:11.901 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' set-attr _ td/osd-unexpected-clone/_ 2026-02-13T02:48:11.901 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-unexpected-clone 2026-02-13T02:48:11.901 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-13T02:48:11.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-13T02:48:11.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-13T02:48:11.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-unexpected-clone/2 2026-02-13T02:48:11.902 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-unexpected-clone/2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' set-attr _ td/osd-unexpected-clone/_ 2026-02-13T02:48:12.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-unexpected-clone 2 2026-02-13T02:48:12.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-unexpected-clone 2026-02-13T02:48:12.188 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-13T02:48:12.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-13T02:48:12.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-13T02:48:12.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-unexpected-clone/2 2026-02-13T02:48:12.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=61ed748e-0c83-461d-a19a-2a41771343bc --auth-supported=none --mon-host=127.0.0.1:7144 ' 2026-02-13T02:48:12.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-13T02:48:12.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-13T02:48:12.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-13T02:48:12.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-unexpected-clone/2' 2026-02-13T02:48:12.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-unexpected-clone/2/journal' 2026-02-13T02:48:12.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-13T02:48:12.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-13T02:48:12.189 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-unexpected-clone' 2026-02-13T02:48:12.190 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-13T02:48:12.190 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-13T02:48:12.190 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-13T02:48:12.190 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-13T02:48:12.190 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:48:12.190 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.653483 2026-02-13T02:48:12.190 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.653483/$cluster-$name.asok' 2026-02-13T02:48:12.190 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.653483/$cluster-$name.asok' 2026-02-13T02:48:12.190 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-13T02:48:12.190 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-unexpected-clone/$name.log' 2026-02-13T02:48:12.190 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-unexpected-clone/$name.pid' 2026-02-13T02:48:12.190 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-13T02:48:12.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-13T02:48:12.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-13T02:48:12.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-13T02:48:12.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-13T02:48:12.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-13T02:48:12.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-unexpected-clone/2 2026-02-13T02:48:12.191 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-13T02:48:12.191 INFO:tasks.workunit.client.0.trial112.stderr:start osd.2 2026-02-13T02:48:12.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=61ed748e-0c83-461d-a19a-2a41771343bc --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/2 --osd-journal=td/osd-unexpected-clone/2/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.653483/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-13T02:48:12.192 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-unexpected-clone/2/whoami 2026-02-13T02:48:12.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-13T02:48:12.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-13T02:48:12.192 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-13T02:48:12.193 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-13T02:48:12.206 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:48:12.202+0000 7f5508d05a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:48:12.207 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:48:12.202+0000 7f5508d05a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:48:12.208 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:48:12.202+0000 7f5508d05a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-13T02:48:12.345 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-13T02:48:12.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-13T02:48:12.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-13T02:48:12.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-13T02:48:12.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-13T02:48:12.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:48:12.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-13T02:48:12.346 INFO:tasks.workunit.client.0.trial112.stderr:0 2026-02-13T02:48:12.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:48:12.346 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:48:12.513 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-13T02:48:12.655 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:48:12.650+0000 7f5508d05a40 -1 Falling back to public interface 2026-02-13T02:48:13.264 INFO:tasks.workunit.client.0.trial112.stderr:2026-02-13T02:48:13.258+0000 7f5508d05a40 -1 osd.2 41 log_to_monitors true 2026-02-13T02:48:13.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-13T02:48:13.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-13T02:48:13.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-13T02:48:13.514 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:48:13.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-13T02:48:13.514 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-13T02:48:13.679 INFO:tasks.workunit.client.0.trial112.stderr:osd.2 up in weight 1 up_from 45 up_thru 0 down_at 42 last_clean_interval [39,41) [v2:127.0.0.1:6818/1277412204,v1:127.0.0.1:6819/1277412204] [v2:127.0.0.1:6820/1277412204,v1:127.0.0.1:6821/1277412204] exists,up 06345ee9-d0aa-4b2e-8a96-42eaf7210230 2026-02-13T02:48:13.680 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-13T02:48:13.680 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-13T02:48:13.680 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-13T02:48:13.680 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-13T02:48:13.680 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-13T02:48:13.680 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-13T02:48:13.680 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-13T02:48:13.680 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-13T02:48:13.680 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-13T02:48:13.680 INFO:tasks.workunit.client.0.trial112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-13T02:48:13.681 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-13T02:48:13.681 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-13T02:48:13.681 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-13T02:48:13.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-13T02:48:13.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-13T02:48:13.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-13T02:48:13.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-13T02:48:13.776 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-13T02:48:13.776 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-13T02:48:13.937 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-13T02:48:13.937 INFO:tasks.workunit.client.0.trial112.stderr:1 2026-02-13T02:48:13.937 INFO:tasks.workunit.client.0.trial112.stderr:2' 2026-02-13T02:48:13.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-13T02:48:13.938 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:48:13.938 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-13T02:48:14.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836495 2026-02-13T02:48:14.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836495 2026-02-13T02:48:14.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836495' 2026-02-13T02:48:14.010 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:48:14.011 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-13T02:48:14.084 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964117001 2026-02-13T02:48:14.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964117001 2026-02-13T02:48:14.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836495 1-115964117001' 2026-02-13T02:48:14.085 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-13T02:48:14.085 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-13T02:48:14.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=193273528322 2026-02-13T02:48:14.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 193273528322 2026-02-13T02:48:14.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836495 1-115964117001 2-193273528322' 2026-02-13T02:48:14.158 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:48:14.159 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836495 2026-02-13T02:48:14.159 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:48:14.160 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-13T02:48:14.161 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836495 2026-02-13T02:48:14.161 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:48:14.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836495 2026-02-13T02:48:14.162 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2026-02-13T02:48:14.162 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.0 seq 21474836495 2026-02-13T02:48:14.162 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-13T02:48:14.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836495 -lt 21474836495 2026-02-13T02:48:14.321 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:48:14.322 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-115964117001 2026-02-13T02:48:14.322 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:48:14.323 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-13T02:48:14.324 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-115964117001 2026-02-13T02:48:14.324 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:48:14.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=115964117001 2026-02-13T02:48:14.326 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 115964117001' 2026-02-13T02:48:14.326 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.1 seq 115964117001 2026-02-13T02:48:14.326 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-13T02:48:14.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 115964117001 -lt 115964117001 2026-02-13T02:48:14.484 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-13T02:48:14.484 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-193273528322 2026-02-13T02:48:14.485 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-13T02:48:14.486 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-13T02:48:14.486 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-193273528322 2026-02-13T02:48:14.486 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-13T02:48:14.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=193273528322 2026-02-13T02:48:14.488 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 193273528322' 2026-02-13T02:48:14.488 INFO:tasks.workunit.client.0.trial112.stderr:waiting osd.2 seq 193273528322 2026-02-13T02:48:14.488 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-13T02:48:14.656 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 193273528322 -lt 193273528322 2026-02-13T02:48:14.657 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-13T02:48:14.657 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:48:14.657 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:48:14.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-13T02:48:14.851 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-13T02:48:14.851 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-13T02:48:14.851 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-13T02:48:14.851 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-13T02:48:14.851 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-13T02:48:14.852 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-13T02:48:14.852 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-13T02:48:15.008 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-13T02:48:15.009 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-13T02:48:15.009 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-13T02:48:15.009 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-13T02:48:15.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-13T02:48:15.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-13T02:48:15.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-13T02:48:15.197 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:70: TEST_recover_unexpected: sleep 5 2026-02-13T02:48:20.198 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:72: TEST_recover_unexpected: ceph pg repair 1.0 2026-02-13T02:48:20.347 INFO:tasks.workunit.client.0.trial112.stderr:instructing pg 1.0 on osd.1 to repair 2026-02-13T02:48:20.358 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:74: TEST_recover_unexpected: sleep 10 2026-02-13T02:48:30.360 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:76: TEST_recover_unexpected: ceph log last 2026-02-13T02:48:30.552 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T02:48:20.698378+0000 osd.1 (osd.1) 3 : cluster [ERR] 1.0 shard 1 1:602f83fe:::foo:1 : missing 2026-02-13T02:48:30.552 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T02:48:20.698460+0000 osd.1 (osd.1) 4 : cluster [ERR] repair 1.0 1:602f83fe:::foo:1 : is an unexpected clone 2026-02-13T02:48:30.552 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T02:48:20.698557+0000 osd.1 (osd.1) 5 : cluster [ERR] 1.0 repair 1 missing, 0 inconsistent objects 2026-02-13T02:48:30.552 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T02:48:20.698566+0000 osd.1 (osd.1) 6 : cluster [ERR] 1.0 repair 1 missing, 0 inconsistent objects 2026-02-13T02:48:30.552 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T02:48:20.698589+0000 osd.1 (osd.1) 7 : cluster [ERR] 1.0 repair 3 errors, 2 fixed 2026-02-13T02:48:30.552 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T02:48:22.219714+0000 mon.a (mon.0) 242 : cluster [ERR] Health check failed: 3 scrub errors (OSD_SCRUB_ERRORS) 2026-02-13T02:48:30.552 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T02:48:22.219737+0000 mon.a (mon.0) 243 : cluster [ERR] Health check failed: Possible data damage: 1 pg inconsistent (PG_DAMAGED) 2026-02-13T02:48:30.552 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T02:48:21.717728+0000 osd.1 (osd.1) 9 : cluster [ERR] 1.0 shard 1 soid 1:602f83fe:::foo:1 : candidate size 0 info size 15806 mismatch 2026-02-13T02:48:30.552 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T02:48:21.717774+0000 osd.1 (osd.1) 10 : cluster [ERR] 1.0 shard 0 soid 1:602f83fe:::foo:1 : data_digest 0xdfb728bf != data_digest 0xf85c6ade from auth oi 1:602f83fe:::foo:1(18'2 client.4156.0:1 dirty|data_digest s 15806 uv 1 dd f85c6ade alloc_hint [0 0 0]) 2026-02-13T02:48:30.553 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T02:48:21.717776+0000 osd.1 (osd.1) 11 : cluster [ERR] 1.0 shard 1 soid 1:602f83fe:::foo:1 : data_digest 0xffffffff != data_digest 0xdfb728bf from shard 0, data_digest 0xffffffff != data_digest 0xf85c6ade from auth oi 1:602f83fe:::foo:1(18'2 client.4156.0:1 dirty|data_digest s 15806 uv 1 dd f85c6ade alloc_hint [0 0 0]), size 0 != size 15806 from auth oi 1:602f83fe:::foo:1(18'2 client.4156.0:1 dirty|data_digest s 15806 uv 1 dd f85c6ade alloc_hint [0 0 0]), size 0 != size 15806 from shard 0 2026-02-13T02:48:30.553 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T02:48:21.717777+0000 osd.1 (osd.1) 12 : cluster [ERR] 1.0 soid 1:602f83fe:::foo:1 : data_digest 0xf85c6ade != data_digest 0xdfb728bf from shard 0 2026-02-13T02:48:30.553 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T02:48:21.717847+0000 osd.1 (osd.1) 13 : cluster [ERR] deep-scrub 1.0 1:602f83fe:::foo:1 : is an unexpected clone 2026-02-13T02:48:30.553 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T02:48:21.718012+0000 osd.1 (osd.1) 14 : cluster [ERR] 1.0 deep-scrub 0 missing, 1 inconsistent objects 2026-02-13T02:48:30.553 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T02:48:21.718015+0000 osd.1 (osd.1) 15 : cluster [ERR] 1.0 deep-scrub 4 errors 2026-02-13T02:48:30.553 INFO:tasks.workunit.client.0.trial112.stdout:2026-02-13T02:48:28.220142+0000 mon.a (mon.0) 244 : cluster [ERR] Health check update: 4 scrub errors (OSD_SCRUB_ERRORS) 2026-02-13T02:48:30.562 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:79: TEST_recover_unexpected: timeout 60 ceph tell osd.0 version 2026-02-13T02:48:30.630 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:48:30.630 INFO:tasks.workunit.client.0.trial112.stdout: "version": "20.2.0-636-g3b40640f", 2026-02-13T02:48:30.630 INFO:tasks.workunit.client.0.trial112.stdout: "release": "tentacle", 2026-02-13T02:48:30.631 INFO:tasks.workunit.client.0.trial112.stdout: "release_type": "stable" 2026-02-13T02:48:30.631 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:48:30.637 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:80: TEST_recover_unexpected: timeout 60 ceph tell osd.1 version 2026-02-13T02:48:30.702 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:48:30.702 INFO:tasks.workunit.client.0.trial112.stdout: "version": "20.2.0-636-g3b40640f", 2026-02-13T02:48:30.702 INFO:tasks.workunit.client.0.trial112.stdout: "release": "tentacle", 2026-02-13T02:48:30.702 INFO:tasks.workunit.client.0.trial112.stdout: "release_type": "stable" 2026-02-13T02:48:30.702 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:48:30.709 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:81: TEST_recover_unexpected: timeout 60 ceph tell osd.2 version 2026-02-13T02:48:30.774 INFO:tasks.workunit.client.0.trial112.stdout:{ 2026-02-13T02:48:30.774 INFO:tasks.workunit.client.0.trial112.stdout: "version": "20.2.0-636-g3b40640f", 2026-02-13T02:48:30.774 INFO:tasks.workunit.client.0.trial112.stdout: "release": "tentacle", 2026-02-13T02:48:30.774 INFO:tasks.workunit.client.0.trial112.stdout: "release_type": "stable" 2026-02-13T02:48:30.774 INFO:tasks.workunit.client.0.trial112.stdout:} 2026-02-13T02:48:30.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:35: run: teardown td/osd-unexpected-clone 2026-02-13T02:48:30.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-unexpected-clone 2026-02-13T02:48:30.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-13T02:48:30.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-unexpected-clone KILL 2026-02-13T02:48:30.783 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:48:30.783 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:48:30.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:48:30.783 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:48:30.784 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:48:30.892 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:48:30.892 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:48:30.893 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:48:30.893 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:48:30.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:48:30.894 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:48:30.895 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:48:30.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:48:30.896 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:48:30.897 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:48:30.897 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:48:30.898 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:48:30.899 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-13T02:48:30.899 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-unexpected-clone 2026-02-13T02:48:30.911 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:48:30.911 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:48:30.911 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.653483 2026-02-13T02:48:30.911 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.653483 2026-02-13T02:48:30.914 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:48:30.914 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:48:30.914 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: main: code=0 2026-02-13T02:48:30.914 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: teardown td/osd-unexpected-clone 0 2026-02-13T02:48:30.914 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-unexpected-clone 2026-02-13T02:48:30.914 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-13T02:48:30.914 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-unexpected-clone KILL 2026-02-13T02:48:30.914 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-13T02:48:30.914 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-13T02:48:30.914 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-13T02:48:30.915 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-13T02:48:30.915 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-13T02:48:30.916 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-13T02:48:30.916 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-13T02:48:30.917 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-13T02:48:30.917 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-13T02:48:30.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-13T02:48:30.918 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-13T02:48:30.918 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-13T02:48:30.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:48:30.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-13T02:48:30.920 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-13T02:48:30.920 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-13T02:48:30.921 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:48:30.922 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-13T02:48:30.923 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-unexpected-clone 2026-02-13T02:48:30.924 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-13T02:48:30.924 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-13T02:48:30.924 INFO:tasks.workunit.client.0.trial112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.653483 2026-02-13T02:48:30.924 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.653483 2026-02-13T02:48:30.925 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-13T02:48:30.925 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-13T02:48:30.925 INFO:tasks.workunit.client.0.trial112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: main: return 0 2026-02-13T02:48:30.927 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-13T02:48:30.927 DEBUG:teuthology.orchestra.run.trial112:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-13T02:48:30.981 INFO:tasks.workunit:Stopping ['scrub'] on client.0... 2026-02-13T02:48:30.982 DEBUG:teuthology.orchestra.run.trial112:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2026-02-13T02:48:31.339 DEBUG:teuthology.parallel:result is None 2026-02-13T02:48:31.340 DEBUG:teuthology.orchestra.run.trial112:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2026-02-13T02:48:31.348 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2026-02-13T02:48:31.348 DEBUG:teuthology.orchestra.run.trial112:> rmdir -- /home/ubuntu/cephtest/mnt.0 2026-02-13T02:48:31.393 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2026-02-13T02:48:31.393 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-13T02:48:31.409 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-13T02:48:31.410 DEBUG:teuthology.orchestra.run.trial112:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-13T02:48:31.542 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-13T02:48:31.554 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-13T02:48:31.554 DEBUG:teuthology.orchestra.run.trial112:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-13T02:48:31.639 INFO:teuthology.orchestra.run.trial112.stdout: remote refid st t when poll reach delay offset jitter 2026-02-13T02:48:31.640 INFO:teuthology.orchestra.run.trial112.stdout:============================================================================== 2026-02-13T02:48:31.640 INFO:teuthology.orchestra.run.trial112.stdout:-ntp.maxhost.io 132.163.97.4 2 u 36 64 377 69.776 +10.849 5.367 2026-02-13T02:48:31.640 INFO:teuthology.orchestra.run.trial112.stdout:+69.164.213.136 97.183.206.88 2 u 50 64 377 3.384 +4.031 1.061 2026-02-13T02:48:31.640 INFO:teuthology.orchestra.run.trial112.stdout:+time-dfw.far.ne 17.253.6.45 2 u 18 64 377 41.445 +0.781 0.536 2026-02-13T02:48:31.640 INFO:teuthology.orchestra.run.trial112.stdout:*chi2.us.ntp.li 204.9.54.119 2 u 31 64 377 19.594 +2.572 0.580 2026-02-13T02:48:31.640 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-13T02:48:31.652 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-13T02:48:31.652 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-13T02:48:31.668 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-13T02:48:31.682 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-13T02:48:31.697 INFO:teuthology.task.internal:Duration was 3447.493550 seconds 2026-02-13T02:48:31.698 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-13T02:48:31.712 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-13T02:48:31.712 DEBUG:teuthology.orchestra.run.trial112:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-13T02:48:31.738 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-13T02:48:31.738 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial112.front.sepia.ceph.com 2026-02-13T02:48:31.738 DEBUG:teuthology.orchestra.run.trial112:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-13T02:48:31.791 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-13T02:48:31.791 DEBUG:teuthology.orchestra.run.trial112:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-13T02:48:31.973 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-13T02:48:31.973 DEBUG:teuthology.orchestra.run.trial112:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-13T02:48:31.982 INFO:teuthology.orchestra.run.trial112.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-13T02:48:31.983 INFO:teuthology.orchestra.run.trial112.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-13T02:48:31.983 INFO:teuthology.orchestra.run.trial112.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-13T02:48:31.983 INFO:teuthology.orchestra.run.trial112.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-13T02:48:31.984 INFO:teuthology.orchestra.run.trial112.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: /home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-13T02:48:32.023 INFO:teuthology.orchestra.run.trial112.stderr: 82.3% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-13T02:48:32.025 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-13T02:48:32.041 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-13T02:48:32.041 DEBUG:teuthology.orchestra.run.trial112:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-13T02:48:32.073 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-13T02:48:32.091 DEBUG:teuthology.orchestra.run.trial112:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:48:32.125 INFO:teuthology.orchestra.run.trial112.stdout:kernel.core_pattern = core 2026-02-13T02:48:32.133 DEBUG:teuthology.orchestra.run.trial112:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-13T02:48:32.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-13T02:48:32.176 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-13T02:48:32.193 INFO:teuthology.task.internal:Transferring archived files... 2026-02-13T02:48:32.195 DEBUG:teuthology.misc:Transferring archived files from trial112:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/skanta-2026-02-13_01:43:11-rados-wip-bharath17-testing-2026-02-12-0618-tentacle-distro-default-trial/47689/remote/trial112 2026-02-13T02:48:32.196 DEBUG:teuthology.orchestra.run.trial112:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-13T02:48:32.244 INFO:teuthology.task.internal:Removing archive directory... 2026-02-13T02:48:32.245 DEBUG:teuthology.orchestra.run.trial112:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-13T02:48:32.281 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-13T02:48:32.298 INFO:teuthology.task.internal:Not uploading archives. 2026-02-13T02:48:32.299 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-13T02:48:32.314 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-13T02:48:32.315 DEBUG:teuthology.orchestra.run.trial112:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-13T02:48:32.325 INFO:teuthology.orchestra.run.trial112.stdout: 1180220 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 13 02:48 /home/ubuntu/cephtest 2026-02-13T02:48:32.326 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-13T02:48:32.343 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-13T02:48:32.379 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub} duration: 3447.4935495853424 flavor: default owner: scheduled_skanta@soko04.front.sepia.ceph.com success: true 2026-02-13T02:48:32.379 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-13T02:48:32.500 INFO:teuthology.run:pass